Jul 28 00:53:37.854207 ip-10-0-138-250 systemd[1]: Starting Kubernetes Kubelet... Jul 28 00:53:38.244400 ip-10-0-138-250 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 00:53:38.244400 ip-10-0-138-250 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 00:53:38.244400 ip-10-0-138-250 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 00:53:38.244400 ip-10-0-138-250 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 00:53:38.244400 ip-10-0-138-250 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 00:53:38.245543 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.245423 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 00:53:38.251148 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251129 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:53:38.251148 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251145 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:53:38.251148 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251150 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:53:38.251148 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251155 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251159 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251164 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251168 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251171 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251176 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251179 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251183 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251187 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251191 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251195 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251199 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251205 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251209 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251220 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251224 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251228 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251232 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251236 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:53:38.251388 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251240 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251244 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251248 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251252 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251257 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251261 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251267 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251273 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251278 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251283 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251287 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251291 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251296 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251302 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251309 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251314 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251319 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251323 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251327 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:53:38.252187 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251332 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251337 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251341 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251345 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251353 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251359 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251364 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251369 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251373 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251378 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251382 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251386 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251390 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251394 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251399 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251403 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251407 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251411 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251416 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251421 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:53:38.253006 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251425 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251430 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251434 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251441 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251446 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251450 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251454 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251459 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251464 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251469 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251475 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251479 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251483 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251488 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251492 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251496 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251501 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251505 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251509 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251513 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:53:38.253895 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251518 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251522 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251526 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251530 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251534 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251538 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251542 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251547 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251551 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251556 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251560 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251564 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251568 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251573 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251576 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251581 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251585 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.251589 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252786 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252798 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:53:38.254768 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252804 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252809 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252813 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252819 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252824 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252828 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252833 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252837 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252842 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252846 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252850 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252854 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252861 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252867 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252871 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252876 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252880 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252884 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252888 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252892 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:53:38.255644 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252896 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252900 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252905 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252909 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252914 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252918 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252922 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252926 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252945 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252950 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252954 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252959 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252965 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252969 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252973 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252977 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252983 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252988 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252992 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.252996 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:53:38.256314 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253000 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253004 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253008 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253012 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253015 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253020 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253025 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253029 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253033 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253037 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253042 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253046 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253050 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253055 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253059 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253063 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253067 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253072 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253076 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253080 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:53:38.256816 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253084 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253088 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253092 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253097 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253102 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253109 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253114 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253119 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253124 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253129 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253134 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253139 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253143 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253148 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253161 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253165 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253170 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253175 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253181 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:53:38.257408 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253187 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253192 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253196 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253200 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253204 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253208 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253212 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253216 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253220 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253224 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253228 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253235 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253241 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253245 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253249 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253253 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253257 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253261 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253874 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:53:38.258118 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253883 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253887 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253892 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253897 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253902 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253907 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253911 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253915 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253919 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253923 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253927 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253955 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253960 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253967 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253971 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253976 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253980 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253984 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253988 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253992 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:53:38.258637 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.253996 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254001 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254005 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254010 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254014 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254018 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254023 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254028 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254032 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254038 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254044 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254050 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254054 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254059 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254067 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254072 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254078 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254082 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254087 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254091 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:53:38.259200 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254095 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254100 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254104 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254108 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254112 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254118 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254122 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254126 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254130 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254136 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254140 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254145 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254149 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254153 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254158 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254162 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254168 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254173 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254178 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:53:38.259780 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254183 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254187 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254192 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254196 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254200 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254205 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254211 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254215 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254219 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254224 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254229 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254233 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254237 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254241 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254245 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254250 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254254 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254258 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254264 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254268 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:53:38.260288 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254272 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254276 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254282 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254287 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254292 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254296 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254300 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254305 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254309 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254313 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254318 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254322 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254326 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254330 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254334 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254339 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254343 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254347 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.254352 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254446 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254457 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 00:53:38.260796 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254466 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254472 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254484 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254490 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254497 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254503 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254508 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254513 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254519 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254525 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254531 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254535 2580 flags.go:64] FLAG: --cgroup-root="" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254540 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254548 2580 flags.go:64] FLAG: --client-ca-file="" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254552 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254557 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254564 2580 flags.go:64] FLAG: --cluster-domain="" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254569 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254574 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254579 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254585 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254591 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254595 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 00:53:38.261317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254600 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254605 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254610 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254615 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254620 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254625 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254629 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254636 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254641 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254646 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254650 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254655 2580 flags.go:64] FLAG: --enable-server="true" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254661 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254667 2580 flags.go:64] FLAG: --event-burst="100" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254672 2580 flags.go:64] FLAG: --event-qps="50" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254676 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254681 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254686 2580 flags.go:64] FLAG: --eviction-hard="" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254695 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254700 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254705 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254710 2580 flags.go:64] FLAG: --eviction-soft="" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254717 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254721 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254726 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 00:53:38.261878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254732 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254737 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254741 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254747 2580 flags.go:64] FLAG: --feature-gates="" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254753 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254757 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254762 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254767 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254771 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254776 2580 flags.go:64] FLAG: --help="false" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254780 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254785 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254790 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254794 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254800 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254805 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254811 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254816 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254820 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254825 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254831 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254837 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254841 2580 flags.go:64] FLAG: --kube-reserved="" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254846 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 00:53:38.262859 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254851 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254856 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254861 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254866 2580 flags.go:64] FLAG: --lock-file="" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254871 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254877 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254882 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254891 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254895 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254900 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254905 2580 flags.go:64] FLAG: --logging-format="text" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254910 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254915 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254920 2580 flags.go:64] FLAG: --manifest-url="" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254924 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254949 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254955 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254961 2580 flags.go:64] FLAG: --max-pods="110" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254966 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254971 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254975 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254980 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254985 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254990 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.254995 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 00:53:38.264015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255006 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255011 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255015 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255020 2580 flags.go:64] FLAG: --pod-cidr="" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255025 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255034 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255039 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255044 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255049 2580 flags.go:64] FLAG: --port="10250" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255055 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255059 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060c837f29677f456" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255065 2580 flags.go:64] FLAG: --qos-reserved="" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255070 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255077 2580 flags.go:64] FLAG: --register-node="true" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255082 2580 flags.go:64] FLAG: --register-with-taints="" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255088 2580 flags.go:64] FLAG: --registry-burst="10" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255093 2580 flags.go:64] FLAG: --registry-qps="5" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255098 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255102 2580 flags.go:64] FLAG: --reserved-memory="" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255108 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255113 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255118 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255123 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255128 2580 flags.go:64] FLAG: --runonce="false" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255133 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 00:53:38.264879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255138 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255143 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255148 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255152 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255157 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255162 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255167 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255172 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255177 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255181 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255186 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255191 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255196 2580 flags.go:64] FLAG: --system-cgroups="" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255201 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255210 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255215 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255219 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255227 2580 flags.go:64] FLAG: --tls-min-version="" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255232 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255236 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255243 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255248 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255253 2580 flags.go:64] FLAG: --v="2" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255265 2580 flags.go:64] FLAG: --version="false" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255272 2580 flags.go:64] FLAG: --vmodule="" Jul 28 00:53:38.265658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255277 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.255283 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255491 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255499 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255504 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255509 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255514 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255519 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255524 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255529 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255533 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255538 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255543 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255548 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255552 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255558 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255563 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255569 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255574 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255579 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:53:38.266876 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255585 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255590 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255594 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255598 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255602 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255606 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255611 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255617 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255621 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255625 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255630 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255634 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255638 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255642 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255646 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255650 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255655 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255658 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255662 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255666 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:53:38.267405 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255671 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255675 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255679 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255683 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255687 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255692 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255696 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255700 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255704 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255708 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255713 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255717 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255721 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255728 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255733 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255737 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255741 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255745 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255749 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255754 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:53:38.268102 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255759 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255763 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255768 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255773 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255777 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255781 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255785 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255789 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255793 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255797 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255802 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255806 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255810 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255814 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255819 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255823 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255827 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255831 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255835 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:53:38.268627 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255841 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255847 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255852 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255857 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255862 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255866 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255873 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255879 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255886 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255891 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255896 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255900 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255906 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255911 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255915 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255919 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255923 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255927 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255947 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255951 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:53:38.269436 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255955 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.255960 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.256831 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.264558 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.264574 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.265337 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.265350 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265441 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265450 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265456 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265460 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265465 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265470 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265474 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265478 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:53:38.270158 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265483 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265487 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265491 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265496 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265502 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265507 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265511 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265517 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265524 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265528 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265533 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265537 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265543 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265548 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265552 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265556 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265560 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265565 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265569 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:53:38.270577 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265573 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265578 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265582 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265589 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265595 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265599 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265604 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265609 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265614 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265618 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265623 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265627 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265632 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265638 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265644 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265648 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265653 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265657 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265662 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:53:38.271144 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265666 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265670 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265674 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265678 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265682 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265687 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265691 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265695 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265700 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265704 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265708 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265713 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265717 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265721 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265725 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265729 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265734 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265738 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265742 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265747 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:53:38.271687 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265751 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265755 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265759 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265763 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265768 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265772 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265776 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265780 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265784 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265789 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265793 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265797 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265803 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265807 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265811 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265815 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265820 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265824 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265829 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265833 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:53:38.272298 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265837 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265842 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265846 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265850 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265854 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265858 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265862 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265867 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265871 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265875 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265879 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265883 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.265888 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.265895 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266128 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:53:38.272800 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266138 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266143 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266151 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266157 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266162 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266167 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266172 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266177 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266182 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266187 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266193 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266199 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266205 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266211 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266216 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266220 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266224 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266229 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266233 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:53:38.273240 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266236 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266240 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266244 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266248 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266252 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266255 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266260 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266264 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266268 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266272 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266276 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266279 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266283 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266287 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266291 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266295 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266298 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266302 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266306 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266310 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:53:38.273726 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266314 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266318 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266322 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266326 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266331 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266335 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266339 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266343 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266347 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266351 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266356 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266360 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266364 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266368 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266373 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266381 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266385 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266388 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266392 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266396 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:53:38.274217 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266403 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266409 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266415 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266420 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266424 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266429 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266433 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266438 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266441 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266446 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266450 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266454 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266459 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266463 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266467 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266471 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266476 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266481 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266485 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266489 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:53:38.274841 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266493 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266497 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266501 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266505 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266510 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266514 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266518 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266524 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266528 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266533 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266537 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266542 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266546 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266551 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266555 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266559 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266563 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266567 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:53:38.275404 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:38.266571 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:53:38.275855 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.266579 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 00:53:38.275855 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.266719 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 00:53:38.275855 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.270592 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 00:53:38.275855 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.271999 2580 server.go:1019] "Starting client certificate rotation" Jul 28 00:53:38.275855 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.272100 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 00:53:38.275855 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.272131 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 00:53:38.277754 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.277728 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 00:53:38.278837 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.278820 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 00:53:38.290754 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.290735 2580 log.go:25] "Validated CRI v1 runtime API" Jul 28 00:53:38.292870 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.292855 2580 log.go:25] "Validated CRI v1 image API" Jul 28 00:53:38.293721 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.293706 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 00:53:38.296172 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.296151 2580 fs.go:135] Filesystem UUIDs: map[152e86e7-a88a-4ca8-a0c2-a9d4340b2851:/dev/nvme0n1p4 660ee9dc-aa40-4e2f-9218-b2357b87ff54:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 28 00:53:38.296249 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.296171 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 00:53:38.300390 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.300279 2580 manager.go:216] Machine: {Timestamp:2026-07-28 00:53:38.299802203 +0000 UTC m=+0.330740435 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100264 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28181c4f3c8bdc0b3d3c504ec09df3 SystemUUID:ec28181c-4f3c-8bdc-0b3d-3c504ec09df3 BootID:8832410b-89a7-4c37-81b2-6e6af4e1f9a6 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:3c:06:7e:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:3c:06:7e:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:9f:34:af:ad:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 00:53:38.300390 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.300380 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 00:53:38.300522 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.300457 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 00:53:38.302191 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.302164 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 00:53:38.302313 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.302193 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 00:53:38.302954 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.302925 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 00:53:38.302997 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.302955 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 00:53:38.302997 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.302990 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 00:53:38.303057 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.302990 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 00:53:38.303689 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.303678 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 00:53:38.304869 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.304858 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 00:53:38.307756 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.307745 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 28 00:53:38.307887 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.307877 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 00:53:38.307948 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.307905 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 28 00:53:38.307948 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.307925 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 00:53:38.308045 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.307961 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 00:53:38.308045 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.307973 2580 kubelet.go:398] "Adding apiserver pod source" Jul 28 00:53:38.308045 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.307982 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 00:53:38.308330 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.308318 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 00:53:38.308361 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.308341 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 00:53:38.310446 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.310432 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 00:53:38.312219 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.312198 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 00:53:38.312295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.312242 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 00:53:38.312995 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.312974 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 00:53:38.312995 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.312999 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 00:53:38.312995 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313007 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 00:53:38.313169 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313013 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 00:53:38.313169 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313022 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 00:53:38.313169 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313028 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 00:53:38.313169 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313155 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 00:53:38.313290 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313181 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 00:53:38.313290 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313190 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 00:53:38.313290 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313197 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 00:53:38.313290 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313226 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 00:53:38.313290 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313249 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 00:53:38.313981 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313970 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 00:53:38.314026 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.313983 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 00:53:38.317834 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.317817 2580 server.go:1268] "Started kubelet" Jul 28 00:53:38.317990 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.317905 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 00:53:38.318400 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.318029 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 00:53:38.318400 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.318089 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 00:53:38.318550 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.318530 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 00:53:38.318622 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.318548 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 00:53:38.318685 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.318673 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 00:53:38.318748 ip-10-0-138-250 systemd[1]: Started Kubernetes Kubelet. Jul 28 00:53:38.319653 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.319635 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 28 00:53:38.322524 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.322506 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 00:53:38.323568 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.322413 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-250.ec2.internal.18c64df550da515e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-250.ec2.internal,UID:ip-10-0-138-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-250.ec2.internal,},FirstTimestamp:2026-07-28 00:53:38.317791582 +0000 UTC m=+0.348729814,LastTimestamp:2026-07-28 00:53:38.317791582 +0000 UTC m=+0.348729814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-250.ec2.internal,}" Jul 28 00:53:38.323881 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.323864 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 00:53:38.328407 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.328383 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 00:53:38.328407 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.328391 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 00:53:38.328533 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.328475 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:38.329033 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329019 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 00:53:38.329146 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329035 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 00:53:38.329221 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329204 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 00:53:38.329221 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329157 2580 factory.go:55] Registering systemd factory Jul 28 00:53:38.329390 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329234 2580 factory.go:223] Registration of the systemd container factory successfully Jul 28 00:53:38.329390 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329288 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 28 00:53:38.329390 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329296 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 28 00:53:38.329523 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329453 2580 factory.go:153] Registering CRI-O factory Jul 28 00:53:38.329523 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329466 2580 factory.go:223] Registration of the crio container factory successfully Jul 28 00:53:38.329523 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329508 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 00:53:38.329630 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329530 2580 factory.go:103] Registering Raw factory Jul 28 00:53:38.329630 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329540 2580 manager.go:1196] Started watching for new ooms in manager Jul 28 00:53:38.329873 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.329862 2580 manager.go:349] Starting recovery of all containers Jul 28 00:53:38.330472 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.330444 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 00:53:38.330637 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.330613 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 00:53:38.335799 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.335775 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vvq5n" Jul 28 00:53:38.339808 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.339786 2580 manager.go:354] Recovery completed Jul 28 00:53:38.343448 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.343391 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vvq5n" Jul 28 00:53:38.345965 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.345925 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:53:38.347732 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.347716 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:53:38.347803 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.347744 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:53:38.347803 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.347754 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:53:38.348290 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.348272 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 00:53:38.348290 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.348285 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:53:38.348412 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.348286 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 00:53:38.348412 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.348328 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 28 00:53:38.348892 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.348878 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:53:38.349034 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.348906 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:53:38.349034 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.348918 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:53:38.349443 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.349429 2580 policy_none.go:49] "None policy: Start" Jul 28 00:53:38.349486 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.349454 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 00:53:38.349486 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.349469 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 00:53:38.350483 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.350472 2580 policy_none.go:47] "Start" Jul 28 00:53:38.390564 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.390447 2580 manager.go:337] "Starting Device Plugin manager" Jul 28 00:53:38.390645 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.390579 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 00:53:38.390645 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.390592 2580 server.go:87] "Starting device plugin registration server" Jul 28 00:53:38.390820 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.390808 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 00:53:38.390873 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.390822 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 00:53:38.390981 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.390927 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 00:53:38.391052 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.391022 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 00:53:38.391052 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.391032 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 00:53:38.391527 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.391505 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 00:53:38.391597 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.391558 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:38.491156 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.491130 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:53:38.492177 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.492100 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:53:38.492177 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.492142 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:53:38.492177 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.492157 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:53:38.492339 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.492196 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.497608 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.497542 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 00:53:38.498751 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.498732 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 00:53:38.498833 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.498770 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 00:53:38.498833 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.498794 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 00:53:38.498906 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.498834 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 00:53:38.501340 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.501322 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:53:38.502251 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.502236 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.502299 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.502258 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-250.ec2.internal\": node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:38.559473 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.559446 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:38.599581 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.599540 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal"] Jul 28 00:53:38.599652 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.599625 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:53:38.600547 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.600530 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:53:38.600621 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.600558 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:53:38.600621 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.600571 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:53:38.601905 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.601892 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:53:38.602033 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.602078 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602049 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:53:38.602568 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602553 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:53:38.602641 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602576 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:53:38.602641 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602587 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:53:38.602641 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602557 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:53:38.602641 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602641 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:53:38.602781 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.602651 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:53:38.603750 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.603737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.603796 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.603762 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:53:38.604340 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.604326 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:53:38.604412 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.604350 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:53:38.604412 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.604364 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:53:38.630442 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.630427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf2ad10d87d1af597abc90248027ab31-config\") pod \"kube-apiserver-proxy-ip-10-0-138-250.ec2.internal\" (UID: \"bf2ad10d87d1af597abc90248027ab31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.630545 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.630447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aadab06d8cfdfbb8e0a5c9918305d6d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal\" (UID: \"aadab06d8cfdfbb8e0a5c9918305d6d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.630545 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:38.630464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aadab06d8cfdfbb8e0a5c9918305d6d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal\" (UID: \"aadab06d8cfdfbb8e0a5c9918305d6d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.630545 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.630494 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-250.ec2.internal\" not found" node="ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.634863 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.634847 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-250.ec2.internal\" not found" node="ip-10-0-138-250.ec2.internal" Jul 28 00:53:38.660483 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:38.660466 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:40.018422 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.018397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aadab06d8cfdfbb8e0a5c9918305d6d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal\" (UID: \"aadab06d8cfdfbb8e0a5c9918305d6d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.018508 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.018432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aadab06d8cfdfbb8e0a5c9918305d6d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal\" (UID: \"aadab06d8cfdfbb8e0a5c9918305d6d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.018508 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.018460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf2ad10d87d1af597abc90248027ab31-config\") pod \"kube-apiserver-proxy-ip-10-0-138-250.ec2.internal\" (UID: \"bf2ad10d87d1af597abc90248027ab31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.018508 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.018495 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf2ad10d87d1af597abc90248027ab31-config\") pod \"kube-apiserver-proxy-ip-10-0-138-250.ec2.internal\" (UID: \"bf2ad10d87d1af597abc90248027ab31\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.018508 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.018497 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aadab06d8cfdfbb8e0a5c9918305d6d4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal\" (UID: \"aadab06d8cfdfbb8e0a5c9918305d6d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.018647 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.018492 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/aadab06d8cfdfbb8e0a5c9918305d6d4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal\" (UID: \"aadab06d8cfdfbb8e0a5c9918305d6d4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.048539 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.048488 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:40.149249 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.149223 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:40.220765 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.220728 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.225047 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.225030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.250359 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.250334 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:40.350902 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.350812 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:40.451305 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.451277 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-250.ec2.internal\" not found" Jul 28 00:53:40.505096 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.505073 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:53:40.512005 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.511987 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:53:40.517037 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.517018 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.527373 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.527343 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 00:53:40.528429 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.528416 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" Jul 28 00:53:40.536123 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.536112 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 00:53:40.559257 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.559241 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 00:53:40.559389 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.559356 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 00:53:40.559445 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.559405 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 00:53:40.559445 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.559422 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 00:53:40.596555 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.596526 2580 apiserver.go:52] "Watching apiserver" Jul 28 00:53:40.613409 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.613341 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 00:53:40.613704 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.613683 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-trvxs","openshift-multus/multus-r7776","openshift-multus/network-metrics-daemon-8b67z","kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal","openshift-cluster-node-tuning-operator/tuned-n5rm2","openshift-dns/node-resolver-6zj2k","openshift-network-diagnostics/network-check-target-5fbnk","openshift-network-operator/iptables-alerter-grsq6","openshift-ovn-kubernetes/ovnkube-node-c28hc","kube-system/global-pull-secret-syncer-f6w4c","kube-system/konnectivity-agent-5jtwb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf","openshift-image-registry/node-ca-rczhf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal"] Jul 28 00:53:40.616111 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.616081 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 00:53:40.616293 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.616269 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.616720 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.616694 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:40.617212 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.617197 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.618345 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.618329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:40.618446 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.618410 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:40.619529 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.619510 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-trkfd\"" Jul 28 00:53:40.619635 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.619618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.619736 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.619720 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 00:53:40.619781 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.619758 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 00:53:40.619781 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.619720 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 00:53:40.620738 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.620722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/321b307e-eb62-4af8-beda-c5bd51c8a9ae-kubelet-config\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.620820 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.620745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlqcr\" (UniqueName: \"kubernetes.io/projected/21bcb53a-99c9-484d-bb0b-f866d79b9db6-kube-api-access-qlqcr\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.620820 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.620764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/321b307e-eb62-4af8-beda-c5bd51c8a9ae-dbus\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.620820 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.620784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.620968 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.620820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21bcb53a-99c9-484d-bb0b-f866d79b9db6-host\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.620968 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.620828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.620968 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.620844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21bcb53a-99c9-484d-bb0b-f866d79b9db6-serviceca\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.621966 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.621948 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.622049 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.622026 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 00:53:40.622299 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.622281 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 00:53:40.622448 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.622431 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z9jfm\"" Jul 28 00:53:40.622494 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.622484 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 00:53:40.623274 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.623254 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h7rjf\"" Jul 28 00:53:40.623274 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.623271 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 00:53:40.623397 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.623282 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 00:53:40.623461 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.623447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.624093 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.624076 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 00:53:40.624584 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.624570 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-49nz8\"" Jul 28 00:53:40.624720 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.624706 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:40.624777 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.624750 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:40.624777 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.624760 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 00:53:40.625732 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.625717 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 00:53:40.625992 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.625976 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 00:53:40.626673 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.626661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7776" Jul 28 00:53:40.626861 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.626768 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 00:53:40.627695 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.627680 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 00:53:40.628012 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.627996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.628192 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.628143 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 00:53:40.628300 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.628173 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 00:53:40.628385 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.628250 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t9w4q\"" Jul 28 00:53:40.628752 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.628735 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 00:53:40.628986 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.628972 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 00:53:40.629025 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.629015 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 00:53:40.629057 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.629032 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 00:53:40.629057 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.629047 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n77xz\"" Jul 28 00:53:40.629971 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.629956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:40.630225 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.630209 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gjx9x\"" Jul 28 00:53:40.630769 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.630756 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 00:53:40.630830 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.630817 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 00:53:40.631334 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.631318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.632348 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.632333 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 00:53:40.632348 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.632342 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 00:53:40.632498 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.632335 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q5z4q\"" Jul 28 00:53:40.633548 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.633531 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 00:53:40.633627 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.633551 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-txw5m\"" Jul 28 00:53:40.633793 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.633778 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 00:53:40.633862 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.633778 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 00:53:40.635278 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.635251 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 00:48:39 +0000 UTC" deadline="2028-02-23 09:08:11.21835579 +0000 UTC" Jul 28 00:53:40.635278 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.635276 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13808h14m30.583082253s" Jul 28 00:53:40.637956 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.637940 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 00:53:40.641583 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.641567 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:53:40.660212 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.660192 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mpxb4" Jul 28 00:53:40.665054 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.665038 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mpxb4" Jul 28 00:53:40.718129 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.718097 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 00:53:40.721428 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721401 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-node-log\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.721543 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721445 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ndc\" (UniqueName: \"kubernetes.io/projected/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-kube-api-access-b9ndc\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.721543 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysconfig\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.721543 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721497 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-var-lib-kubelet\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.721543 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721523 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-var-lib-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwbm\" (UniqueName: \"kubernetes.io/projected/bbddc02b-2a1d-45f1-a400-0be44410b2a9-kube-api-access-hjwbm\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721602 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-systemd-units\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-ovnkube-config\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-cni-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106e16e6-9f66-45e8-b3a6-f23afd18a49e-cni-binary-copy\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721690 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/297a3b21-7219-4065-9ef1-0180fd91da16-konnectivity-ca\") pod \"konnectivity-agent-5jtwb\" (UID: \"297a3b21-7219-4065-9ef1-0180fd91da16\") " pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:40.721737 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721724 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b76a-5d37-4008-91da-f37004123d1b-tmp-dir\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-kubernetes\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysctl-conf\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-sys\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721905 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-slash\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721931 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-system-cni-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/321b307e-eb62-4af8-beda-c5bd51c8a9ae-kubelet-config\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.721988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkk4\" (UniqueName: \"kubernetes.io/projected/aa58ed3c-0998-420d-83f1-82a206ed6cd5-kube-api-access-9dkk4\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-ovn\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-log-socket\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722068 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/321b307e-eb62-4af8-beda-c5bd51c8a9ae-kubelet-config\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/297a3b21-7219-4065-9ef1-0180fd91da16-agent-certs\") pod \"konnectivity-agent-5jtwb\" (UID: \"297a3b21-7219-4065-9ef1-0180fd91da16\") " pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:40.722143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-run\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0afac767-a39e-4ed7-be4e-d1641fde9296-ovn-node-metrics-cert\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-host\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-kubelet\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-run-ovn-kubernetes\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-cni-bin\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722295 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-hostroot\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21bcb53a-99c9-484d-bb0b-f866d79b9db6-host\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21bcb53a-99c9-484d-bb0b-f866d79b9db6-serviceca\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/21bcb53a-99c9-484d-bb0b-f866d79b9db6-host\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722475 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-lib-modules\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.722546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722535 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-k8s-cni-cncf-io\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-socket-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cnibin\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-etc-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722713 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-env-overrides\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-sys-fs\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722758 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysctl-d\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722787 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/321b307e-eb62-4af8-beda-c5bd51c8a9ae-dbus\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722812 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbddc02b-2a1d-45f1-a400-0be44410b2a9-iptables-alerter-script\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbddc02b-2a1d-45f1-a400-0be44410b2a9-host-slash\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-netns\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8edefeef-be1e-494a-a33c-58f18f4f9223-tmp\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4pp\" (UniqueName: \"kubernetes.io/projected/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-kube-api-access-vs4pp\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722946 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/21bcb53a-99c9-484d-bb0b-f866d79b9db6-serviceca\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.723149 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-ovnkube-script-lib\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-conf-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.722964 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/321b307e-eb62-4af8-beda-c5bd51c8a9ae-dbus\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r92tk\" (UniqueName: \"kubernetes.io/projected/6a03b76a-5d37-4008-91da-f37004123d1b-kube-api-access-r92tk\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-cni-netd\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-os-release\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-cni-bin\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-device-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.723220 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbns\" (UniqueName: \"kubernetes.io/projected/8edefeef-be1e-494a-a33c-58f18f4f9223-kube-api-access-hhbns\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hnmg\" (UniqueName: \"kubernetes.io/projected/0afac767-a39e-4ed7-be4e-d1641fde9296-kube-api-access-6hnmg\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.723324 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret podName:321b307e-eb62-4af8-beda-c5bd51c8a9ae nodeName:}" failed. No retries permitted until 2026-07-28 00:53:41.223299387 +0000 UTC m=+1.966806869 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret") pod "global-pull-secret-syncer-f6w4c" (UID: "321b307e-eb62-4af8-beda-c5bd51c8a9ae") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-cnibin\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.723816 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-socket-dir-parent\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-multus-certs\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-modprobe-d\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723472 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a03b76a-5d37-4008-91da-f37004123d1b-hosts-file\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-system-cni-dir\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-run-netns\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-systemd\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-kubelet\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-daemon-config\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlqcr\" (UniqueName: \"kubernetes.io/projected/21bcb53a-99c9-484d-bb0b-f866d79b9db6-kube-api-access-qlqcr\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-tuned\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723687 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-registration-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-os-release\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cni-binary-copy\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-cni-multus\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-etc-kubernetes\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.724437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723810 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqtdx\" (UniqueName: \"kubernetes.io/projected/106e16e6-9f66-45e8-b3a6-f23afd18a49e-kube-api-access-mqtdx\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.724973 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.723851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-systemd\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.729612 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.729575 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 00:53:40.732453 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.732438 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlqcr\" (UniqueName: \"kubernetes.io/projected/21bcb53a-99c9-484d-bb0b-f866d79b9db6-kube-api-access-qlqcr\") pod \"node-ca-rczhf\" (UID: \"21bcb53a-99c9-484d-bb0b-f866d79b9db6\") " pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.804885 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:40.804847 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadab06d8cfdfbb8e0a5c9918305d6d4.slice/crio-309c4a984953bc05a5159e0bf2a49187f37850f05ba33d5b1fa60cb0edcd33f7 WatchSource:0}: Error finding container 309c4a984953bc05a5159e0bf2a49187f37850f05ba33d5b1fa60cb0edcd33f7: Status 404 returned error can't find the container with id 309c4a984953bc05a5159e0bf2a49187f37850f05ba33d5b1fa60cb0edcd33f7 Jul 28 00:53:40.805295 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:40.805278 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2ad10d87d1af597abc90248027ab31.slice/crio-742b388b5fe07969bab1a2fae3556648556ed403bd9d8d01e07a7e667f2dea2c WatchSource:0}: Error finding container 742b388b5fe07969bab1a2fae3556648556ed403bd9d8d01e07a7e667f2dea2c: Status 404 returned error can't find the container with id 742b388b5fe07969bab1a2fae3556648556ed403bd9d8d01e07a7e667f2dea2c Jul 28 00:53:40.809621 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.809602 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:53:40.824986 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.824962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbns\" (UniqueName: \"kubernetes.io/projected/8edefeef-be1e-494a-a33c-58f18f4f9223-kube-api-access-hhbns\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.825084 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.824999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.825084 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825024 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hnmg\" (UniqueName: \"kubernetes.io/projected/0afac767-a39e-4ed7-be4e-d1641fde9296-kube-api-access-6hnmg\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.825193 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-cnibin\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825193 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-socket-dir-parent\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825193 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825138 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-multus-certs\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825193 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-modprobe-d\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.825193 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a03b76a-5d37-4008-91da-f37004123d1b-hosts-file\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-system-cni-dir\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825221 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-multus-certs\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825231 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-cnibin\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-socket-dir-parent\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-run-netns\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825279 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-systemd\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6a03b76a-5d37-4008-91da-f37004123d1b-hosts-file\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-modprobe-d\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-systemd\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825324 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-system-cni-dir\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-kubelet\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825342 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-run-netns\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-daemon-config\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-tuned\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825429 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-kubelet\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.825450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-registration-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-os-release\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cni-binary-copy\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-cni-multus\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-etc-kubernetes\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-registration-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqtdx\" (UniqueName: \"kubernetes.io/projected/106e16e6-9f66-45e8-b3a6-f23afd18a49e-kube-api-access-mqtdx\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825640 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-systemd\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-node-log\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ndc\" (UniqueName: \"kubernetes.io/projected/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-kube-api-access-b9ndc\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825694 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825712 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysconfig\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825728 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-os-release\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-var-lib-kubelet\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825765 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-var-lib-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825763 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-cni-multus\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-node-log\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.826173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825831 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-var-lib-kubelet\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwbm\" (UniqueName: \"kubernetes.io/projected/bbddc02b-2a1d-45f1-a400-0be44410b2a9-kube-api-access-hjwbm\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-systemd-units\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825883 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-systemd\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-ovnkube-config\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825937 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-daemon-config\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-cni-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825773 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-etc-kubernetes\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.825970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106e16e6-9f66-45e8-b3a6-f23afd18a49e-cni-binary-copy\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826040 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysconfig\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/297a3b21-7219-4065-9ef1-0180fd91da16-konnectivity-ca\") pod \"konnectivity-agent-5jtwb\" (UID: \"297a3b21-7219-4065-9ef1-0180fd91da16\") " pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b76a-5d37-4008-91da-f37004123d1b-tmp-dir\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-kubernetes\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysctl-conf\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-sys\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-slash\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-system-cni-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.827004 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826283 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826322 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-var-lib-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826331 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkk4\" (UniqueName: \"kubernetes.io/projected/aa58ed3c-0998-420d-83f1-82a206ed6cd5-kube-api-access-9dkk4\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826386 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-ovn\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-log-socket\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826437 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/297a3b21-7219-4065-9ef1-0180fd91da16-agent-certs\") pod \"konnectivity-agent-5jtwb\" (UID: \"297a3b21-7219-4065-9ef1-0180fd91da16\") " pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826437 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-run\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0afac767-a39e-4ed7-be4e-d1641fde9296-ovn-node-metrics-cert\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-host\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-ovnkube-config\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-kubelet\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-run-ovn-kubernetes\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826574 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a03b76a-5d37-4008-91da-f37004123d1b-tmp-dir\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-cni-bin\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-ovn\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-kubernetes\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826588 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/106e16e6-9f66-45e8-b3a6-f23afd18a49e-cni-binary-copy\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.827931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826094 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-cni-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826609 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-hostroot\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826654 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-log-socket\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826657 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-kubelet\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cni-binary-copy\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826682 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826691 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-run-ovn-kubernetes\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826706 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-systemd-units\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826748 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-host\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-tuning-conf-dir\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826793 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-run\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-hostroot\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826836 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-system-cni-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.826907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827054 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysctl-conf\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.828782 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827069 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/297a3b21-7219-4065-9ef1-0180fd91da16-konnectivity-ca\") pod \"konnectivity-agent-5jtwb\" (UID: \"297a3b21-7219-4065-9ef1-0180fd91da16\") " pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-lib-modules\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827113 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-sys\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827116 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-slash\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827133 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-cni-bin\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-lib-modules\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-k8s-cni-cncf-io\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-socket-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827272 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-socket-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827300 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827312 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-k8s-cni-cncf-io\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cnibin\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827412 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-cnibin\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-etc-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-etc-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.829599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-env-overrides\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-sys-fs\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827535 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysctl-d\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbddc02b-2a1d-45f1-a400-0be44410b2a9-iptables-alerter-script\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827620 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbddc02b-2a1d-45f1-a400-0be44410b2a9-host-slash\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-netns\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8edefeef-be1e-494a-a33c-58f18f4f9223-tmp\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827682 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-sysctl-d\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4pp\" (UniqueName: \"kubernetes.io/projected/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-kube-api-access-vs4pp\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827762 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-ovnkube-script-lib\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-conf-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-env-overrides\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r92tk\" (UniqueName: \"kubernetes.io/projected/6a03b76a-5d37-4008-91da-f37004123d1b-kube-api-access-r92tk\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-cni-netd\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-os-release\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830062 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-cni-bin\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827926 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-device-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-run-netns\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828005 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-device-dir\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828024 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bbddc02b-2a1d-45f1-a400-0be44410b2a9-host-slash\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828117 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-host-cni-netd\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828182 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-os-release\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8edefeef-be1e-494a-a33c-58f18f4f9223-etc-tuned\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828220 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bbddc02b-2a1d-45f1-a400-0be44410b2a9-iptables-alerter-script\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0afac767-a39e-4ed7-be4e-d1641fde9296-run-openvswitch\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.827554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-sys-fs\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828287 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-host-var-lib-cni-bin\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.828293 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828304 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/106e16e6-9f66-45e8-b3a6-f23afd18a49e-multus-conf-dir\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.828531 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs podName:aa58ed3c-0998-420d-83f1-82a206ed6cd5 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:41.328356517 +0000 UTC m=+2.071864003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs") pod "network-metrics-daemon-8b67z" (UID: "aa58ed3c-0998-420d-83f1-82a206ed6cd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.828703 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0afac767-a39e-4ed7-be4e-d1641fde9296-ovnkube-script-lib\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.829031 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0afac767-a39e-4ed7-be4e-d1641fde9296-ovn-node-metrics-cert\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.830574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.829832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/297a3b21-7219-4065-9ef1-0180fd91da16-agent-certs\") pod \"konnectivity-agent-5jtwb\" (UID: \"297a3b21-7219-4065-9ef1-0180fd91da16\") " pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:40.831036 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.829966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8edefeef-be1e-494a-a33c-58f18f4f9223-tmp\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.832602 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.832523 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:53:40.832602 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.832547 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:53:40.832602 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.832560 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v2lvx for pod openshift-network-diagnostics/network-check-target-5fbnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:40.832890 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:40.832628 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx podName:4a864ab0-5e7a-4aa3-bc10-64c02367f7d1 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:41.332611683 +0000 UTC m=+2.076119193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v2lvx" (UniqueName: "kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx") pod "network-check-target-5fbnk" (UID: "4a864ab0-5e7a-4aa3-bc10-64c02367f7d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:40.833510 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.833488 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqtdx\" (UniqueName: \"kubernetes.io/projected/106e16e6-9f66-45e8-b3a6-f23afd18a49e-kube-api-access-mqtdx\") pod \"multus-r7776\" (UID: \"106e16e6-9f66-45e8-b3a6-f23afd18a49e\") " pod="openshift-multus/multus-r7776" Jul 28 00:53:40.833963 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.833934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwbm\" (UniqueName: \"kubernetes.io/projected/bbddc02b-2a1d-45f1-a400-0be44410b2a9-kube-api-access-hjwbm\") pod \"iptables-alerter-grsq6\" (UID: \"bbddc02b-2a1d-45f1-a400-0be44410b2a9\") " pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.833963 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.833952 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbns\" (UniqueName: \"kubernetes.io/projected/8edefeef-be1e-494a-a33c-58f18f4f9223-kube-api-access-hhbns\") pod \"tuned-n5rm2\" (UID: \"8edefeef-be1e-494a-a33c-58f18f4f9223\") " pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.834069 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.834017 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ndc\" (UniqueName: \"kubernetes.io/projected/26e978f8-e6c8-4415-b7b7-e8283fcc83c8-kube-api-access-b9ndc\") pod \"aws-ebs-csi-driver-node-jc7rf\" (UID: \"26e978f8-e6c8-4415-b7b7-e8283fcc83c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:40.834237 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.834222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkk4\" (UniqueName: \"kubernetes.io/projected/aa58ed3c-0998-420d-83f1-82a206ed6cd5-kube-api-access-9dkk4\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:40.834641 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.834621 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hnmg\" (UniqueName: \"kubernetes.io/projected/0afac767-a39e-4ed7-be4e-d1641fde9296-kube-api-access-6hnmg\") pod \"ovnkube-node-c28hc\" (UID: \"0afac767-a39e-4ed7-be4e-d1641fde9296\") " pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:40.835533 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.835517 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4pp\" (UniqueName: \"kubernetes.io/projected/fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916-kube-api-access-vs4pp\") pod \"multus-additional-cni-plugins-trvxs\" (UID: \"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916\") " pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:40.835899 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.835878 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r92tk\" (UniqueName: \"kubernetes.io/projected/6a03b76a-5d37-4008-91da-f37004123d1b-kube-api-access-r92tk\") pod \"node-resolver-6zj2k\" (UID: \"6a03b76a-5d37-4008-91da-f37004123d1b\") " pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.942841 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.942759 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rczhf" Jul 28 00:53:40.951322 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:40.951298 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bcb53a_99c9_484d_bb0b_f866d79b9db6.slice/crio-c724e25c35503a969da2571f9efee373518e7b086649b58c15bc5c778f7eaad6 WatchSource:0}: Error finding container c724e25c35503a969da2571f9efee373518e7b086649b58c15bc5c778f7eaad6: Status 404 returned error can't find the container with id c724e25c35503a969da2571f9efee373518e7b086649b58c15bc5c778f7eaad6 Jul 28 00:53:40.974539 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.974517 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-grsq6" Jul 28 00:53:40.980979 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:40.980956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbddc02b_2a1d_45f1_a400_0be44410b2a9.slice/crio-c15a731be8e17f76a6aefa3ff6b853b5ffe871f526b740afeabe8560b9b8e012 WatchSource:0}: Error finding container c15a731be8e17f76a6aefa3ff6b853b5ffe871f526b740afeabe8560b9b8e012: Status 404 returned error can't find the container with id c15a731be8e17f76a6aefa3ff6b853b5ffe871f526b740afeabe8560b9b8e012 Jul 28 00:53:40.989948 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.989931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" Jul 28 00:53:40.992594 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:40.992580 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6zj2k" Jul 28 00:53:40.998057 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:40.998039 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edefeef_be1e_494a_a33c_58f18f4f9223.slice/crio-7709a7cb16cc9997aa2de468c4d0924c0681d504912060bff27f0e4cb582f3ff WatchSource:0}: Error finding container 7709a7cb16cc9997aa2de468c4d0924c0681d504912060bff27f0e4cb582f3ff: Status 404 returned error can't find the container with id 7709a7cb16cc9997aa2de468c4d0924c0681d504912060bff27f0e4cb582f3ff Jul 28 00:53:41.001604 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:41.001581 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a03b76a_5d37_4008_91da_f37004123d1b.slice/crio-62b0d87d4885f682aee11a471eedadcc898ae054c219b90d8c99d008909f5bd0 WatchSource:0}: Error finding container 62b0d87d4885f682aee11a471eedadcc898ae054c219b90d8c99d008909f5bd0: Status 404 returned error can't find the container with id 62b0d87d4885f682aee11a471eedadcc898ae054c219b90d8c99d008909f5bd0 Jul 28 00:53:41.011265 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.011249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:53:41.019929 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:41.019909 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afac767_a39e_4ed7_be4e_d1641fde9296.slice/crio-c9ce72bd2a3192bd4ee5cafa381a6eba3b26cd1bb90ff4f7490f4e1157f79efb WatchSource:0}: Error finding container c9ce72bd2a3192bd4ee5cafa381a6eba3b26cd1bb90ff4f7490f4e1157f79efb: Status 404 returned error can't find the container with id c9ce72bd2a3192bd4ee5cafa381a6eba3b26cd1bb90ff4f7490f4e1157f79efb Jul 28 00:53:41.025875 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.025849 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r7776" Jul 28 00:53:41.032074 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:41.032052 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod106e16e6_9f66_45e8_b3a6_f23afd18a49e.slice/crio-91ce6f65e868f89ee26dee496055eb9d3e8171795f6fd08e11cd37ccb25d8846 WatchSource:0}: Error finding container 91ce6f65e868f89ee26dee496055eb9d3e8171795f6fd08e11cd37ccb25d8846: Status 404 returned error can't find the container with id 91ce6f65e868f89ee26dee496055eb9d3e8171795f6fd08e11cd37ccb25d8846 Jul 28 00:53:41.045639 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.045624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-trvxs" Jul 28 00:53:41.052002 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:41.051982 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb75e0fc_c01d_4c4d_ae2c_a4c4b9c4d916.slice/crio-77ab69c70cd6bc4708fc86045c9f049905c84145f9cc62acfb5b96cba0b245a0 WatchSource:0}: Error finding container 77ab69c70cd6bc4708fc86045c9f049905c84145f9cc62acfb5b96cba0b245a0: Status 404 returned error can't find the container with id 77ab69c70cd6bc4708fc86045c9f049905c84145f9cc62acfb5b96cba0b245a0 Jul 28 00:53:41.059513 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.059495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:53:41.063469 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.063451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" Jul 28 00:53:41.066352 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:41.066334 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod297a3b21_7219_4065_9ef1_0180fd91da16.slice/crio-833ca3bcb5e97a075582c247ee2538f41efb381c49c124a6c41be17913a117cb WatchSource:0}: Error finding container 833ca3bcb5e97a075582c247ee2538f41efb381c49c124a6c41be17913a117cb: Status 404 returned error can't find the container with id 833ca3bcb5e97a075582c247ee2538f41efb381c49c124a6c41be17913a117cb Jul 28 00:53:41.071121 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:53:41.071102 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e978f8_e6c8_4415_b7b7_e8283fcc83c8.slice/crio-ddcb7a70154c93451e764417840c212f183bf3f8455c76882729bbf7abbf2d97 WatchSource:0}: Error finding container ddcb7a70154c93451e764417840c212f183bf3f8455c76882729bbf7abbf2d97: Status 404 returned error can't find the container with id ddcb7a70154c93451e764417840c212f183bf3f8455c76882729bbf7abbf2d97 Jul 28 00:53:41.230596 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.230506 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:41.230763 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.230644 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:41.230763 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.230714 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret podName:321b307e-eb62-4af8-beda-c5bd51c8a9ae nodeName:}" failed. No retries permitted until 2026-07-28 00:53:42.230695054 +0000 UTC m=+2.974202539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret") pod "global-pull-secret-syncer-f6w4c" (UID: "321b307e-eb62-4af8-beda-c5bd51c8a9ae") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:41.331294 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.331260 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:41.331475 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.331457 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:41.331538 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.331520 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs podName:aa58ed3c-0998-420d-83f1-82a206ed6cd5 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:42.331501793 +0000 UTC m=+3.075009280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs") pod "network-metrics-daemon-8b67z" (UID: "aa58ed3c-0998-420d-83f1-82a206ed6cd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:41.416276 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.416245 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:53:41.432579 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.432548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:41.432756 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.432739 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:53:41.432832 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.432769 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:53:41.432832 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.432783 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v2lvx for pod openshift-network-diagnostics/network-check-target-5fbnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:41.432939 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.432852 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx podName:4a864ab0-5e7a-4aa3-bc10-64c02367f7d1 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:42.432834308 +0000 UTC m=+3.176341809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2lvx" (UniqueName: "kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx") pod "network-check-target-5fbnk" (UID: "4a864ab0-5e7a-4aa3-bc10-64c02367f7d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:41.667232 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.667134 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:48:40 +0000 UTC" deadline="2028-04-06 19:43:42.03748836 +0000 UTC" Jul 28 00:53:41.667232 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.667175 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14850h50m0.370318273s" Jul 28 00:53:41.800678 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.800627 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:41.800855 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.800739 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:41.800855 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.800817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:41.800967 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:41.800912 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:41.823876 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.823648 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"c9ce72bd2a3192bd4ee5cafa381a6eba3b26cd1bb90ff4f7490f4e1157f79efb"} Jul 28 00:53:41.834639 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.834530 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zj2k" event={"ID":"6a03b76a-5d37-4008-91da-f37004123d1b","Type":"ContainerStarted","Data":"62b0d87d4885f682aee11a471eedadcc898ae054c219b90d8c99d008909f5bd0"} Jul 28 00:53:41.858530 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.858495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rczhf" event={"ID":"21bcb53a-99c9-484d-bb0b-f866d79b9db6","Type":"ContainerStarted","Data":"c724e25c35503a969da2571f9efee373518e7b086649b58c15bc5c778f7eaad6"} Jul 28 00:53:41.884071 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.884044 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" event={"ID":"aadab06d8cfdfbb8e0a5c9918305d6d4","Type":"ContainerStarted","Data":"309c4a984953bc05a5159e0bf2a49187f37850f05ba33d5b1fa60cb0edcd33f7"} Jul 28 00:53:41.912546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.912506 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" event={"ID":"26e978f8-e6c8-4415-b7b7-e8283fcc83c8","Type":"ContainerStarted","Data":"ddcb7a70154c93451e764417840c212f183bf3f8455c76882729bbf7abbf2d97"} Jul 28 00:53:41.937554 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.937433 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerStarted","Data":"77ab69c70cd6bc4708fc86045c9f049905c84145f9cc62acfb5b96cba0b245a0"} Jul 28 00:53:41.940401 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.940314 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7776" event={"ID":"106e16e6-9f66-45e8-b3a6-f23afd18a49e","Type":"ContainerStarted","Data":"91ce6f65e868f89ee26dee496055eb9d3e8171795f6fd08e11cd37ccb25d8846"} Jul 28 00:53:41.957473 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.957390 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" event={"ID":"8edefeef-be1e-494a-a33c-58f18f4f9223","Type":"ContainerStarted","Data":"7709a7cb16cc9997aa2de468c4d0924c0681d504912060bff27f0e4cb582f3ff"} Jul 28 00:53:41.970915 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.970812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-grsq6" event={"ID":"bbddc02b-2a1d-45f1-a400-0be44410b2a9","Type":"ContainerStarted","Data":"c15a731be8e17f76a6aefa3ff6b853b5ffe871f526b740afeabe8560b9b8e012"} Jul 28 00:53:41.981575 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.980229 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" event={"ID":"bf2ad10d87d1af597abc90248027ab31","Type":"ContainerStarted","Data":"742b388b5fe07969bab1a2fae3556648556ed403bd9d8d01e07a7e667f2dea2c"} Jul 28 00:53:41.996914 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:41.996878 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jtwb" event={"ID":"297a3b21-7219-4065-9ef1-0180fd91da16","Type":"ContainerStarted","Data":"833ca3bcb5e97a075582c247ee2538f41efb381c49c124a6c41be17913a117cb"} Jul 28 00:53:42.239092 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:42.239013 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:42.239301 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.239173 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:42.239301 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.239252 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret podName:321b307e-eb62-4af8-beda-c5bd51c8a9ae nodeName:}" failed. No retries permitted until 2026-07-28 00:53:44.239232066 +0000 UTC m=+4.982739552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret") pod "global-pull-secret-syncer-f6w4c" (UID: "321b307e-eb62-4af8-beda-c5bd51c8a9ae") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:42.340935 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:42.340176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:42.340935 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.340422 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:42.340935 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.340511 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs podName:aa58ed3c-0998-420d-83f1-82a206ed6cd5 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:44.340491148 +0000 UTC m=+5.083998637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs") pod "network-metrics-daemon-8b67z" (UID: "aa58ed3c-0998-420d-83f1-82a206ed6cd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:42.441519 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:42.440914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:42.441519 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.441083 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:53:42.441519 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.441102 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:53:42.441519 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.441115 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v2lvx for pod openshift-network-diagnostics/network-check-target-5fbnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:42.441519 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.441171 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx podName:4a864ab0-5e7a-4aa3-bc10-64c02367f7d1 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:44.441152324 +0000 UTC m=+5.184659823 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2lvx" (UniqueName: "kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx") pod "network-check-target-5fbnk" (UID: "4a864ab0-5e7a-4aa3-bc10-64c02367f7d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:42.668244 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:42.668130 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:48:40 +0000 UTC" deadline="2028-04-21 00:12:30.089685412 +0000 UTC" Jul 28 00:53:42.668244 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:42.668194 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15191h18m47.421494357s" Jul 28 00:53:42.779878 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:42.779838 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:53:42.789386 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:42.786713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:42.789386 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:42.786814 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:43.460165 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:43.460061 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:53:43.790946 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:43.789836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:43.790946 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:43.789964 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:43.790946 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:43.790447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:43.790946 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:43.790553 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:44.257389 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:44.256424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:44.257389 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.256541 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:44.257389 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.256589 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret podName:321b307e-eb62-4af8-beda-c5bd51c8a9ae nodeName:}" failed. No retries permitted until 2026-07-28 00:53:48.256575311 +0000 UTC m=+9.000082793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret") pod "global-pull-secret-syncer-f6w4c" (UID: "321b307e-eb62-4af8-beda-c5bd51c8a9ae") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:44.359703 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:44.359664 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:44.359879 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.359858 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:44.359960 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.359926 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs podName:aa58ed3c-0998-420d-83f1-82a206ed6cd5 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:48.359908215 +0000 UTC m=+9.103415700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs") pod "network-metrics-daemon-8b67z" (UID: "aa58ed3c-0998-420d-83f1-82a206ed6cd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:44.460384 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:44.460325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:44.460572 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.460518 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:53:44.460572 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.460539 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:53:44.460572 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.460554 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v2lvx for pod openshift-network-diagnostics/network-check-target-5fbnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:44.460737 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.460616 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx podName:4a864ab0-5e7a-4aa3-bc10-64c02367f7d1 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:48.460596035 +0000 UTC m=+9.204103520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2lvx" (UniqueName: "kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx") pod "network-check-target-5fbnk" (UID: "4a864ab0-5e7a-4aa3-bc10-64c02367f7d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:44.787119 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:44.787087 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:44.787281 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:44.787227 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:45.786477 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:45.786441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:45.786962 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:45.786570 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:45.787045 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:45.786961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:45.787094 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:45.787074 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:46.787275 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:46.787230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:46.787764 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:46.787418 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:47.787008 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:47.786974 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:47.787159 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:47.786992 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:47.787159 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:47.787106 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:47.787159 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:47.787129 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:48.290177 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:48.290127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:48.290650 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.290278 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:48.290650 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.290339 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret podName:321b307e-eb62-4af8-beda-c5bd51c8a9ae nodeName:}" failed. No retries permitted until 2026-07-28 00:53:56.290320782 +0000 UTC m=+17.033828279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret") pod "global-pull-secret-syncer-f6w4c" (UID: "321b307e-eb62-4af8-beda-c5bd51c8a9ae") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:48.392197 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:48.391510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:48.392197 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.391695 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:48.392197 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.391759 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs podName:aa58ed3c-0998-420d-83f1-82a206ed6cd5 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:56.39173979 +0000 UTC m=+17.135247276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs") pod "network-metrics-daemon-8b67z" (UID: "aa58ed3c-0998-420d-83f1-82a206ed6cd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:48.492520 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:48.492457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:48.492684 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.492626 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:53:48.492684 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.492648 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:53:48.492684 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.492662 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v2lvx for pod openshift-network-diagnostics/network-check-target-5fbnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:48.492862 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.492727 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx podName:4a864ab0-5e7a-4aa3-bc10-64c02367f7d1 nodeName:}" failed. No retries permitted until 2026-07-28 00:53:56.492706447 +0000 UTC m=+17.236213946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2lvx" (UniqueName: "kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx") pod "network-check-target-5fbnk" (UID: "4a864ab0-5e7a-4aa3-bc10-64c02367f7d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:48.786710 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:48.786593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:48.786846 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:48.786730 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:49.788246 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:49.788187 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:49.788744 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:49.788324 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:49.788744 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:49.788393 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:49.788744 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:49.788500 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:50.786904 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:50.786867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:50.787099 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:50.787023 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:51.789713 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:51.789684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:51.790235 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:51.789684 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:51.790235 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:51.789809 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:51.790235 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:51.789903 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:52.786702 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:52.786614 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:52.786878 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:52.786748 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:53.787256 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:53.787220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:53.787666 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:53.787224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:53.787666 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:53.787347 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:53.787666 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:53.787399 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:54.786872 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:54.786840 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:54.787027 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:54.786958 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:55.786649 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:55.786617 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:55.787075 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:55.786618 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:55.787075 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:55.786750 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:55.787075 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:55.786825 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:56.352813 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:56.352768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:56.353017 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.352915 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:56.353017 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.352996 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret podName:321b307e-eb62-4af8-beda-c5bd51c8a9ae nodeName:}" failed. No retries permitted until 2026-07-28 00:54:12.352978417 +0000 UTC m=+33.096485912 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret") pod "global-pull-secret-syncer-f6w4c" (UID: "321b307e-eb62-4af8-beda-c5bd51c8a9ae") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:53:56.453940 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:56.453905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:56.454090 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.454058 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:56.454157 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.454146 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs podName:aa58ed3c-0998-420d-83f1-82a206ed6cd5 nodeName:}" failed. No retries permitted until 2026-07-28 00:54:12.45413031 +0000 UTC m=+33.197637808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs") pod "network-metrics-daemon-8b67z" (UID: "aa58ed3c-0998-420d-83f1-82a206ed6cd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:53:56.554291 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:56.554207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:56.554472 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.554388 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:53:56.554472 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.554408 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:53:56.554472 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.554418 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v2lvx for pod openshift-network-diagnostics/network-check-target-5fbnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:56.554629 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.554474 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx podName:4a864ab0-5e7a-4aa3-bc10-64c02367f7d1 nodeName:}" failed. No retries permitted until 2026-07-28 00:54:12.554455732 +0000 UTC m=+33.297963223 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2lvx" (UniqueName: "kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx") pod "network-check-target-5fbnk" (UID: "4a864ab0-5e7a-4aa3-bc10-64c02367f7d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:53:56.787229 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:56.787188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:56.787702 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:56.787344 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:57.787300 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:57.787265 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:57.787757 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:57.787413 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:53:57.787757 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:57.787457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:57.787757 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:57.787573 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:58.786995 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:58.786959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:53:58.787175 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:58.787098 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:53:59.787469 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:59.787246 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:53:59.787900 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:53:59.787317 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:53:59.787900 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:59.787536 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:53:59.787900 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:53:59.787608 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:00.786954 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:00.786933 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:00.787061 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:00.787038 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:54:01.039271 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.039198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r7776" event={"ID":"106e16e6-9f66-45e8-b3a6-f23afd18a49e","Type":"ContainerStarted","Data":"3c9a3117d338aedee33b0041ac771cda85b6d47e4d1bd6aa608b9c2176e90abc"} Jul 28 00:54:01.047487 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.047161 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" event={"ID":"8edefeef-be1e-494a-a33c-58f18f4f9223","Type":"ContainerStarted","Data":"982ade9778159690e99a87bfa53c2a4d5bba200c9489970db13dc41bafdbbe52"} Jul 28 00:54:01.056812 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.056626 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r7776" podStartSLOduration=2.146946385 podStartE2EDuration="22.056587341s" totalImagesPullingTime="19.909640956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:01.055550625 +0000 UTC m=+21.799058129" watchObservedRunningTime="2026-07-28 00:54:01.056587341 +0000 UTC m=+21.800094847" Jul 28 00:54:01.060135 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.060029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" event={"ID":"bf2ad10d87d1af597abc90248027ab31","Type":"ContainerStarted","Data":"8e214c99d0d364ac55563cfe51f885f0f2cfc9423c75d8244d77900f4c987ac6"} Jul 28 00:54:01.071116 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.071075 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n5rm2" podStartSLOduration=2.4943521730000002 podStartE2EDuration="22.071057031s" totalImagesPullingTime="19.576704858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:01.070430407 +0000 UTC m=+21.813937911" watchObservedRunningTime="2026-07-28 00:54:01.071057031 +0000 UTC m=+21.814564540" Jul 28 00:54:01.071897 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.071876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 00:54:01.073759 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.073731 2580 generic.go:356] "Generic (PLEG): container finished" podID="0afac767-a39e-4ed7-be4e-d1641fde9296" containerID="a972cce9f3cbab4cabd687e139979ae85dc7ea35293cd141f63f3acaec78fe09" exitCode=1 Jul 28 00:54:01.073833 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.073778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"c98fdde4f53b9bc6e540757d5c48d09a53275517082190991355aea61940fa70"} Jul 28 00:54:01.073876 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.073854 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerDied","Data":"a972cce9f3cbab4cabd687e139979ae85dc7ea35293cd141f63f3acaec78fe09"} Jul 28 00:54:01.073927 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.073909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"e25f62031814d23d509832b1b791516468a8d17561b4e546e4b753004b7e2086"} Jul 28 00:54:01.787419 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.787386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:01.787586 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:01.787386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:01.787586 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:01.787475 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:01.787586 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:01.787542 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:02.077261 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.077221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5jtwb" event={"ID":"297a3b21-7219-4065-9ef1-0180fd91da16","Type":"ContainerStarted","Data":"ba4857eac5e0bf452bf9e153a57ec002993d8b100e0ff97712ce58523319df77"} Jul 28 00:54:02.082436 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.082417 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 00:54:02.083008 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.082983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"f325242f07088f54f88791f9fec212cacaaab83b7a3dfe82deb5647aa402a022"} Jul 28 00:54:02.083120 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.083062 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"51d9fd6a527bcb04d2825b1f8959be782a281776f6298bb0b81b6166c997324d"} Jul 28 00:54:02.083120 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.083109 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"b594e413334dd4f99bc9659336c81b6cebe300663a096bf3966d27ad3d2c45a1"} Jul 28 00:54:02.084462 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.084439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6zj2k" event={"ID":"6a03b76a-5d37-4008-91da-f37004123d1b","Type":"ContainerStarted","Data":"d9e7ec5f48d7dbd4765059df63460f4c217db30595807f0acdaf17e3986a6cad"} Jul 28 00:54:02.085843 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.085820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rczhf" event={"ID":"21bcb53a-99c9-484d-bb0b-f866d79b9db6","Type":"ContainerStarted","Data":"5359451426c3e0a086777361802fcb913925c47261e3d6ad1d09acf038cd63d7"} Jul 28 00:54:02.086845 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.086828 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:54:02.087961 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.087940 2580 generic.go:356] "Generic (PLEG): container finished" podID="aadab06d8cfdfbb8e0a5c9918305d6d4" containerID="326246f2497635f2cc3c2b394191899fcd1e2303889dd6f737ab11d5390b1234" exitCode=0 Jul 28 00:54:02.088056 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.088012 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" event={"ID":"aadab06d8cfdfbb8e0a5c9918305d6d4","Type":"ContainerDied","Data":"326246f2497635f2cc3c2b394191899fcd1e2303889dd6f737ab11d5390b1234"} Jul 28 00:54:02.089495 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.089474 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" event={"ID":"26e978f8-e6c8-4415-b7b7-e8283fcc83c8","Type":"ContainerStarted","Data":"88072c6f929793d837ebd543760ee3b3f4e7107b521ee157fda34fdd7560d6bf"} Jul 28 00:54:02.090927 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.090903 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916" containerID="57772456e9da686479bfd865d0cc6b263f380c73fe921867cfd2dabe0fd3ecba" exitCode=0 Jul 28 00:54:02.091017 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.090969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerDied","Data":"57772456e9da686479bfd865d0cc6b263f380c73fe921867cfd2dabe0fd3ecba"} Jul 28 00:54:02.091285 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.091243 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-250.ec2.internal" podStartSLOduration=22.09122859 podStartE2EDuration="22.09122859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:01.082996918 +0000 UTC m=+21.826504423" watchObservedRunningTime="2026-07-28 00:54:02.09122859 +0000 UTC m=+22.834736099" Jul 28 00:54:02.091829 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.091699 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5jtwb" podStartSLOduration=3.587012982 podStartE2EDuration="23.091687965s" totalImagesPullingTime="19.504674983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:02.090835539 +0000 UTC m=+22.834343044" watchObservedRunningTime="2026-07-28 00:54:02.091687965 +0000 UTC m=+22.835195469" Jul 28 00:54:02.092503 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.092482 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-grsq6" event={"ID":"bbddc02b-2a1d-45f1-a400-0be44410b2a9","Type":"ContainerStarted","Data":"78cb85c325fba58bc38ef1d9096617f78f4be31c01a8ae0fb81a897e4dedd628"} Jul 28 00:54:02.118132 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.118078 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rczhf" podStartSLOduration=7.8561559469999995 podStartE2EDuration="23.118065089s" totalImagesPullingTime="15.261909142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:02.117709233 +0000 UTC m=+22.861216738" watchObservedRunningTime="2026-07-28 00:54:02.118065089 +0000 UTC m=+22.861572592" Jul 28 00:54:02.120231 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.120206 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:02.130524 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.130479 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6zj2k" podStartSLOduration=7.918229487 podStartE2EDuration="23.130471177s" totalImagesPullingTime="15.21224169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:02.130303402 +0000 UTC m=+22.873810907" watchObservedRunningTime="2026-07-28 00:54:02.130471177 +0000 UTC m=+22.873978681" Jul 28 00:54:02.146631 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.146583 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-grsq6" podStartSLOduration=3.553808341 podStartE2EDuration="23.146570914s" totalImagesPullingTime="19.592762573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:02.146453487 +0000 UTC m=+22.889961007" watchObservedRunningTime="2026-07-28 00:54:02.146570914 +0000 UTC m=+22.890078417" Jul 28 00:54:02.267325 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.267162 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:02.367092 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.367062 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:02.522553 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.522525 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:02.573661 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.573606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:02.624043 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.624018 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 00:54:02.693042 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.692945 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T00:54:02.624039223Z","UUID":"8baeab08-373b-4a62-8c3b-13c1d77ff684","Handler":null,"Name":"","Endpoint":""} Jul 28 00:54:02.694982 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.694813 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 00:54:02.694982 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.694853 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 00:54:02.786921 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:02.786891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:02.787120 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:02.787021 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:54:03.097327 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:03.097262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" event={"ID":"aadab06d8cfdfbb8e0a5c9918305d6d4","Type":"ContainerStarted","Data":"56f25c7425765eb6554a984577b23158bfbcdc0e34482b21efef5100947c4187"} Jul 28 00:54:03.099371 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:03.099340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" event={"ID":"26e978f8-e6c8-4415-b7b7-e8283fcc83c8","Type":"ContainerStarted","Data":"53b7646c079d2f4509f17d03d3bd169420441946f82e53ba9c92a9083fb33398"} Jul 28 00:54:03.112120 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:03.112076 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-250.ec2.internal" podStartSLOduration=23.112063021 podStartE2EDuration="23.112063021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:03.111460105 +0000 UTC m=+23.854967609" watchObservedRunningTime="2026-07-28 00:54:03.112063021 +0000 UTC m=+23.855570526" Jul 28 00:54:03.491879 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:03.491847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:03.787496 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:03.787398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:03.787496 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:03.787453 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:03.787718 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:03.787545 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:03.787718 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:03.787673 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:04.105172 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:04.105095 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 00:54:04.105644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:04.105620 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"79a33e7db20506c4d3afccad253bad26a8dd4acd94f3ed10063cd28b805af8a1"} Jul 28 00:54:04.108336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:04.108307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" event={"ID":"26e978f8-e6c8-4415-b7b7-e8283fcc83c8","Type":"ContainerStarted","Data":"8aa0013aef4023a37395fbb21aa9c95e6fa6b64c8ee4f063ea8f32ddd35adcb6"} Jul 28 00:54:04.125095 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:04.125050 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jc7rf" podStartSLOduration=3.109925753 podStartE2EDuration="25.125035483s" totalImagesPullingTime="22.01510973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:04.124900754 +0000 UTC m=+24.868408295" watchObservedRunningTime="2026-07-28 00:54:04.125035483 +0000 UTC m=+24.868542987" Jul 28 00:54:04.787311 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:04.787275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:04.787520 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:04.787440 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:54:05.049802 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:05.049717 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:54:05.050382 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:05.050350 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:54:05.111415 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:05.111383 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5jtwb" Jul 28 00:54:05.169306 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:05.169274 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:05.787097 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:05.787062 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:05.787268 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:05.787196 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:05.787268 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:05.787242 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:05.787427 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:05.787376 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:06.787292 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:06.787059 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:06.787926 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:06.787343 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:54:07.117764 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.117697 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 00:54:07.118090 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.118069 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"6fb5eb063d92a283c0b19529c16a7cf926e745487c3444cdbd0b2346b45ba3fe"} Jul 28 00:54:07.118462 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.118416 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:54:07.118577 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.118496 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:54:07.118577 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.118513 2580 scope.go:117] "RemoveContainer" containerID="a972cce9f3cbab4cabd687e139979ae85dc7ea35293cd141f63f3acaec78fe09" Jul 28 00:54:07.119921 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.119896 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916" containerID="b005c98077abc2345eb38b898fdc763d54b2dbf4917ff3415eabfa8707e4c15f" exitCode=0 Jul 28 00:54:07.120011 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.119992 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerDied","Data":"b005c98077abc2345eb38b898fdc763d54b2dbf4917ff3415eabfa8707e4c15f"} Jul 28 00:54:07.134495 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.134471 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:54:07.787220 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.787189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:07.787439 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:07.787286 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:07.787439 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:07.787346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:07.787923 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:07.787467 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:08.126005 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.125924 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 00:54:08.126338 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.126310 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" event={"ID":"0afac767-a39e-4ed7-be4e-d1641fde9296","Type":"ContainerStarted","Data":"9ba6977401a0851995b8cebaca784c0def68f4101e0ab9cbc105b17759f3ef92"} Jul 28 00:54:08.126700 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.126592 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:54:08.142588 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.142557 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:54:08.152589 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.152534 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" podStartSLOduration=9.554917813 podStartE2EDuration="29.152514452s" totalImagesPullingTime="19.597596639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:08.150793205 +0000 UTC m=+28.894300714" watchObservedRunningTime="2026-07-28 00:54:08.152514452 +0000 UTC m=+28.896021958" Jul 28 00:54:08.484435 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.484400 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5fbnk"] Jul 28 00:54:08.484584 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.484524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:08.484662 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:08.484637 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:08.487396 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.487354 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6w4c"] Jul 28 00:54:08.487572 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.487472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:08.487638 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:08.487573 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:08.488730 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.488706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8b67z"] Jul 28 00:54:08.488830 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.488815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:08.488942 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:08.488924 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:54:08.991266 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:08.991226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:09.130902 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:09.130869 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916" containerID="231d0ad450eb429f36791cadc75e076933444a10a93915c39be333f49c60f4db" exitCode=0 Jul 28 00:54:09.131056 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:09.130948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerDied","Data":"231d0ad450eb429f36791cadc75e076933444a10a93915c39be333f49c60f4db"} Jul 28 00:54:09.583006 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:09.582961 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:09.663379 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:09.663333 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:09.726765 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:09.726737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:09.781349 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:09.781314 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:09.788739 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:09.788708 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:09.788916 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:09.788825 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:10.458948 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:10.458908 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:10.549186 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:10.549113 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6zj2k_6a03b76a-5d37-4008-91da-f37004123d1b/dns-node-resolver/0.log" Jul 28 00:54:10.559106 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:10.559080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:10.786606 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:10.786572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:10.786775 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:10.786615 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:10.786775 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:10.786682 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:54:10.786846 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:10.786787 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:10.890738 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:10.890657 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:10.941383 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:10.941348 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:11.137499 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.137465 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916" containerID="0890be244ee1e88428490f6a1671572f14dd3a7a6b8f02b59b62255010894152" exitCode=0 Jul 28 00:54:11.137634 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.137528 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerDied","Data":"0890be244ee1e88428490f6a1671572f14dd3a7a6b8f02b59b62255010894152"} Jul 28 00:54:11.321330 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.321297 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:11.377781 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.377749 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:11.599418 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.599333 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:11.699619 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.699587 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:11.789493 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.789466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:11.789639 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:11.789561 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:11.949575 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:11.949504 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rczhf_21bcb53a-99c9-484d-bb0b-f866d79b9db6/node-ca/0.log" Jul 28 00:54:12.354734 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.354628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:12.371971 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.371939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:12.372127 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.372093 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:54:12.372181 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.372159 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret podName:321b307e-eb62-4af8-beda-c5bd51c8a9ae nodeName:}" failed. No retries permitted until 2026-07-28 00:54:44.372141365 +0000 UTC m=+65.115648854 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret") pod "global-pull-secret-syncer-f6w4c" (UID: "321b307e-eb62-4af8-beda-c5bd51c8a9ae") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:54:12.455778 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.455742 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:12.472643 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.472615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:12.472813 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.472727 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:54:12.472897 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.472828 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs podName:aa58ed3c-0998-420d-83f1-82a206ed6cd5 nodeName:}" failed. No retries permitted until 2026-07-28 00:54:44.47281087 +0000 UTC m=+65.216318367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs") pod "network-metrics-daemon-8b67z" (UID: "aa58ed3c-0998-420d-83f1-82a206ed6cd5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:54:12.573549 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.573511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:12.573724 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.573684 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:54:12.573724 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.573707 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:54:12.573724 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.573721 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v2lvx for pod openshift-network-diagnostics/network-check-target-5fbnk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:54:12.573862 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.573790 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx podName:4a864ab0-5e7a-4aa3-bc10-64c02367f7d1 nodeName:}" failed. No retries permitted until 2026-07-28 00:54:44.57376964 +0000 UTC m=+65.317277135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v2lvx" (UniqueName: "kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx") pod "network-check-target-5fbnk" (UID: "4a864ab0-5e7a-4aa3-bc10-64c02367f7d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:54:12.578734 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.578703 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:12.679187 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.679076 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:12.726502 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.726466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:12.787453 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.787415 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:12.787453 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.787447 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:12.787694 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.787545 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fbnk" podUID="4a864ab0-5e7a-4aa3-bc10-64c02367f7d1" Jul 28 00:54:12.787694 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:12.787674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8b67z" podUID="aa58ed3c-0998-420d-83f1-82a206ed6cd5" Jul 28 00:54:12.999166 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:12.999087 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:13.095143 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:13.095112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:13.244905 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:13.244840 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:13.299656 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:13.299579 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:13.555796 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:13.555695 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:13.655840 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:13.655797 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:13.757237 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:13.757198 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:13.789252 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:13.789223 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:13.789404 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:13.789323 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f6w4c" podUID="321b307e-eb62-4af8-beda-c5bd51c8a9ae" Jul 28 00:54:14.127767 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.127732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:14.595529 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.595498 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-250.ec2.internal" event="NodeReady" Jul 28 00:54:14.595700 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.595660 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 00:54:14.786657 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.786567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:14.787091 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.786891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:14.789427 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.789400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 00:54:14.789551 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.789431 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 00:54:14.789551 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.789432 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wktlp\"" Jul 28 00:54:14.789648 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.789557 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 00:54:14.789701 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:14.789663 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cknlx\"" Jul 28 00:54:15.430709 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:15.430454 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:15.518136 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:15.518113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:15.789358 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:15.789295 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:15.791921 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:15.791901 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 00:54:15.820333 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:15.820311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:15.928962 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:15.928935 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:16.084921 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.084835 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:16.151477 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.151448 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916" containerID="287f81cab779964f3fc111e21f1c279d404e4b0f48f1a87d8aacdc2ddf0c9419" exitCode=0 Jul 28 00:54:16.151619 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.151492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerDied","Data":"287f81cab779964f3fc111e21f1c279d404e4b0f48f1a87d8aacdc2ddf0c9419"} Jul 28 00:54:16.176584 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.176558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:16.232030 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.232008 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:16.400769 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.400680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:16.500922 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.500886 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:16.814064 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:16.813995 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:17.156611 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:17.156539 2580 generic.go:356] "Generic (PLEG): container finished" podID="fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916" containerID="c19c677f57b728e4b7e1c95343e7bddbe19ae96be9353aa616987597ea51d123" exitCode=0 Jul 28 00:54:17.156611 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:17.156578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerDied","Data":"c19c677f57b728e4b7e1c95343e7bddbe19ae96be9353aa616987597ea51d123"} Jul 28 00:54:17.411829 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:17.411739 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:18.013665 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:18.013622 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:18.114955 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:18.114878 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:18.162007 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:18.161978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-trvxs" event={"ID":"fb75e0fc-c01d-4c4d-ae2c-a4c4b9c4d916","Type":"ContainerStarted","Data":"0247be5a147461498ee5d00f3c07d758d7029528f38231097ab3b83d470b64ad"} Jul 28 00:54:18.182262 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:18.182217 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-trvxs" podStartSLOduration=8.784109028 podStartE2EDuration="39.182200271s" totalImagesPullingTime="30.398091243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:18.181157438 +0000 UTC m=+38.924664942" watchObservedRunningTime="2026-07-28 00:54:18.182200271 +0000 UTC m=+38.925707771" Jul 28 00:54:18.206820 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:18.206793 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:18.609347 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:18.609253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:18.822262 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:18.822225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:19.160988 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:19.160918 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:19.261118 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:19.261068 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:19.616751 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:19.616671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:19.719261 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:19.719216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:20.213625 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:20.213596 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:20.314889 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:20.314863 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:20.415185 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:20.415155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:20.516017 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:20.515944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:20.616262 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:20.616231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:20.716716 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:20.716686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:20.818179 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:20.818096 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.025166 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.025135 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.125588 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.125520 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.225842 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.225813 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.276580 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.276552 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.539090 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.539014 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.639965 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.639931 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.740383 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.740334 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.841574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.841505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:21.941494 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:21.941464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:22.090384 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:22.090336 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:22.206345 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:22.206268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:22.308540 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:22.308511 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:22.608615 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:22.608545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:23.010333 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:23.010257 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:23.407593 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:23.407520 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:23.612076 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:23.612048 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:23.712612 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:23.712535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:23.813926 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:23.813897 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:24.215057 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:24.215018 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:24.609142 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:24.609065 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:24.806215 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:24.806185 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:25.215045 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:25.215013 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:25.608145 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:25.608119 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:26.007906 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:26.007818 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:26.209485 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:26.209451 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:26.408530 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:26.408502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:26.605196 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:26.605168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:27.362954 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:27.362925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:27.451965 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:27.451930 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:28.504333 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:28.504303 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:28.551332 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:28.551306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:31.084074 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.084044 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-775fbd7745-kgh54"] Jul 28 00:54:31.098445 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.098420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.100542 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.100516 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 00:54:31.100678 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.100546 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xbmf2\"" Jul 28 00:54:31.101542 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.101513 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nk2n7"] Jul 28 00:54:31.102125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.102028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 00:54:31.102125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.102066 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 00:54:31.106558 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.106538 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 00:54:31.114382 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.114343 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-775fbd7745-kgh54"] Jul 28 00:54:31.114482 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.114389 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-twvxb"] Jul 28 00:54:31.115088 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.114612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.116789 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.116768 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 00:54:31.117070 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.117045 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 00:54:31.117168 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.117111 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r566t\"" Jul 28 00:54:31.117168 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.117119 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 00:54:31.117168 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.117141 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 00:54:31.134600 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.134578 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-775fbd7745-kgh54"] Jul 28 00:54:31.134681 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.134611 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nk2n7"] Jul 28 00:54:31.134681 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.134636 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-twvxb"] Jul 28 00:54:31.134751 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.134703 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.134833 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:31.134799 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lzss5 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-lzss5 registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-775fbd7745-kgh54" podUID="d7c79a1c-6dde-4e5a-8963-8623d9459941" Jul 28 00:54:31.137085 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.137065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w6lql\"" Jul 28 00:54:31.137183 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.137107 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 00:54:31.137183 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.137154 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 00:54:31.188059 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.188032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:31.189217 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.189202 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.209518 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.209488 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-txb7z"] Jul 28 00:54:31.210345 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c79a1c-6dde-4e5a-8963-8623d9459941-ca-trust-extracted\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.210424 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.210424 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vj82\" (UniqueName: \"kubernetes.io/projected/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-kube-api-access-6vj82\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.210424 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-image-registry-private-configuration\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.210517 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-tls\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.210517 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-data-volume\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.210517 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-certificates\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.210517 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-bound-sa-token\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.210634 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzss5\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-kube-api-access-lzss5\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.210634 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.210634 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-config-volume\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.210634 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210630 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-metrics-tls\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.210753 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-tmp-dir\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.210753 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwnj\" (UniqueName: \"kubernetes.io/projected/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-kube-api-access-pzwnj\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.210753 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210737 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-crio-socket\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.210843 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-trusted-ca\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.210843 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.210779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-installation-pull-secrets\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.212458 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.212434 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.222563 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.222539 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l6mrb"] Jul 28 00:54:31.222669 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.222653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-txb7z" Jul 28 00:54:31.225620 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.225602 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-zjtxh\"" Jul 28 00:54:31.225796 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.225782 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 00:54:31.225999 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.225983 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 00:54:31.236517 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.236494 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh"] Jul 28 00:54:31.236675 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.236653 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.239582 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.239558 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s4cm9\"" Jul 28 00:54:31.239741 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.239715 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 00:54:31.240178 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.240156 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 00:54:31.240273 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.240219 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 00:54:31.256511 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.256490 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k"] Jul 28 00:54:31.256619 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.256601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.259015 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.258996 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 00:54:31.259111 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.259035 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 00:54:31.259111 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.259105 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 00:54:31.259200 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.259105 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 28 00:54:31.259200 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.259136 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w2pmq\"" Jul 28 00:54:31.272552 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.272532 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8"] Jul 28 00:54:31.272650 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.272636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.275198 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.275181 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 28 00:54:31.293421 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.293350 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:31.294724 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.294709 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fd8dbc94d-4df67"] Jul 28 00:54:31.294834 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.294820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.296853 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.296838 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 28 00:54:31.297037 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.297022 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 28 00:54:31.297744 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.297726 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 28 00:54:31.297966 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.297948 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 28 00:54:31.311898 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.311875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-bound-sa-token\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312033 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.311910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzss5\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-kube-api-access-lzss5\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312033 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.311929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.312033 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.311948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-config-volume\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.312033 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.311970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-metrics-tls\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.312240 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-tmp-dir\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.312240 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312123 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwnj\" (UniqueName: \"kubernetes.io/projected/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-kube-api-access-pzwnj\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.312240 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-crio-socket\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.312240 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjqmn\" (UniqueName: \"kubernetes.io/projected/fd775c74-88f4-4574-9c7b-eae74a9fb32a-kube-api-access-bjqmn\") pod \"downloads-7c97dc4555-txb7z\" (UID: \"fd775c74-88f4-4574-9c7b-eae74a9fb32a\") " pod="openshift-console/downloads-7c97dc4555-txb7z" Jul 28 00:54:31.312240 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-trusted-ca\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-installation-pull-secrets\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312332 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c79a1c-6dde-4e5a-8963-8623d9459941-ca-trust-extracted\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vj82\" (UniqueName: \"kubernetes.io/projected/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-kube-api-access-6vj82\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312428 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-image-registry-private-configuration\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312453 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-tls\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312474 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-data-volume\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312473 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-tmp-dir\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.312515 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312497 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-certificates\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.312924 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312610 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-config-volume\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.312924 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.312613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.313036 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.313018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-crio-socket\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.313432 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.313414 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-data-volume\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.313546 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.313132 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c79a1c-6dde-4e5a-8963-8623d9459941-ca-trust-extracted\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.313833 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.313800 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-certificates\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.313833 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.313814 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-trusted-ca\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.314796 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.314774 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-txb7z"] Jul 28 00:54:31.314955 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.314936 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k"] Jul 28 00:54:31.314955 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.314968 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh"] Jul 28 00:54:31.315112 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.314866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.315112 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.314982 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6mrb"] Jul 28 00:54:31.315112 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.315000 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd8dbc94d-4df67"] Jul 28 00:54:31.315112 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.315016 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8"] Jul 28 00:54:31.316704 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.316674 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.316798 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.316778 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-image-registry-private-configuration\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.316847 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.316812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-metrics-tls\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.316883 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.316865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-tls\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.316883 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.316877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-installation-pull-secrets\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.333655 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.333635 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwnj\" (UniqueName: \"kubernetes.io/projected/edfa93cf-d7aa-4ba6-a01d-c60611cdfb34-kube-api-access-pzwnj\") pod \"dns-default-twvxb\" (UID: \"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34\") " pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.348659 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.348633 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-bound-sa-token\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.348765 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.348662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzss5\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-kube-api-access-lzss5\") pod \"image-registry-775fbd7745-kgh54\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:31.348811 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.348770 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vj82\" (UniqueName: \"kubernetes.io/projected/9e5a3055-47c7-4ccc-9907-a7c96db9b9a6-kube-api-access-6vj82\") pod \"insights-runtime-extractor-nk2n7\" (UID: \"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6\") " pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.389123 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.389090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:31.413795 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.413769 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-tls\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.413908 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.413809 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-installation-pull-secrets\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.413908 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.413847 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c79a1c-6dde-4e5a-8963-8623d9459941-ca-trust-extracted\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.413908 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.413864 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-image-registry-private-configuration\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.413908 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.413882 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-certificates\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.414052 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.413930 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-trusted-ca\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.414052 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414026 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-bound-sa-token\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.414136 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-registry-tls\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.414136 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbaf0501-e6b9-46f1-a830-53b1a41dd7ab-cert\") pod \"ingress-canary-l6mrb\" (UID: \"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab\") " pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.414136 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414110 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/224dea5b-9272-45de-a02b-2a6c556e445d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.414281 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.414281 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414154 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c79a1c-6dde-4e5a-8963-8623d9459941-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:31.414281 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpld7\" (UniqueName: \"kubernetes.io/projected/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-kube-api-access-xpld7\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.414281 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06587788-5cdc-4c3a-be0b-21dc0414aee8-trusted-ca\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.414505 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdck\" (UniqueName: \"kubernetes.io/projected/dbaf0501-e6b9-46f1-a830-53b1a41dd7ab-kube-api-access-9xdck\") pod \"ingress-canary-l6mrb\" (UID: \"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab\") " pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.414505 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-klusterlet-config\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.414505 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-ca\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.414505 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c27888c0-7a26-4595-8baa-8ea243216c6b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-856b685fd-cqhkh\" (UID: \"c27888c0-7a26-4595-8baa-8ea243216c6b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.414505 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414493 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:54:31.414699 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414522 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:54:31.414699 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414518 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06587788-5cdc-4c3a-be0b-21dc0414aee8-registry-certificates\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.414699 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-tmp\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.414699 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414597 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.414699 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p89kz\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-kube-api-access-p89kz\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.414699 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjqmn\" (UniqueName: \"kubernetes.io/projected/fd775c74-88f4-4574-9c7b-eae74a9fb32a-kube-api-access-bjqmn\") pod \"downloads-7c97dc4555-txb7z\" (UID: \"fd775c74-88f4-4574-9c7b-eae74a9fb32a\") " pod="openshift-console/downloads-7c97dc4555-txb7z" Jul 28 00:54:31.414928 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-hub\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.414928 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wcdt\" (UniqueName: \"kubernetes.io/projected/224dea5b-9272-45de-a02b-2a6c556e445d-kube-api-access-5wcdt\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.414928 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414762 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06587788-5cdc-4c3a-be0b-21dc0414aee8-image-registry-private-configuration\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.414928 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06587788-5cdc-4c3a-be0b-21dc0414aee8-installation-pull-secrets\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.414928 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhw7j\" (UniqueName: \"kubernetes.io/projected/c27888c0-7a26-4595-8baa-8ea243216c6b-kube-api-access-nhw7j\") pod \"managed-serviceaccount-addon-agent-856b685fd-cqhkh\" (UID: \"c27888c0-7a26-4595-8baa-8ea243216c6b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.414928 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06587788-5cdc-4c3a-be0b-21dc0414aee8-ca-trust-extracted\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.415112 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414942 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-trusted-ca\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.415112 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414958 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d7c79a1c-6dde-4e5a-8963-8623d9459941-ca-trust-extracted\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.415112 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.414973 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-certificates\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.416142 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.416123 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:54:31.416353 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.416340 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:31.416418 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.416351 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:54:31.422505 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.422487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjqmn\" (UniqueName: \"kubernetes.io/projected/fd775c74-88f4-4574-9c7b-eae74a9fb32a-kube-api-access-bjqmn\") pod \"downloads-7c97dc4555-txb7z\" (UID: \"fd775c74-88f4-4574-9c7b-eae74a9fb32a\") " pod="openshift-console/downloads-7c97dc4555-txb7z" Jul 28 00:54:31.424278 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.424263 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nk2n7" Jul 28 00:54:31.442561 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.442540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:31.487808 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.487776 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:31.516075 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516008 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-bound-sa-token\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.516075 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516063 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzss5\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-kube-api-access-lzss5\") pod \"d7c79a1c-6dde-4e5a-8963-8623d9459941\" (UID: \"d7c79a1c-6dde-4e5a-8963-8623d9459941\") " Jul 28 00:54:31.516319 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wcdt\" (UniqueName: \"kubernetes.io/projected/224dea5b-9272-45de-a02b-2a6c556e445d-kube-api-access-5wcdt\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.516319 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06587788-5cdc-4c3a-be0b-21dc0414aee8-image-registry-private-configuration\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.516319 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06587788-5cdc-4c3a-be0b-21dc0414aee8-installation-pull-secrets\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.516319 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhw7j\" (UniqueName: \"kubernetes.io/projected/c27888c0-7a26-4595-8baa-8ea243216c6b-kube-api-access-nhw7j\") pod \"managed-serviceaccount-addon-agent-856b685fd-cqhkh\" (UID: \"c27888c0-7a26-4595-8baa-8ea243216c6b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.516319 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06587788-5cdc-4c3a-be0b-21dc0414aee8-ca-trust-extracted\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.516319 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-bound-sa-token\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-registry-tls\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbaf0501-e6b9-46f1-a830-53b1a41dd7ab-cert\") pod \"ingress-canary-l6mrb\" (UID: \"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab\") " pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/224dea5b-9272-45de-a02b-2a6c556e445d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpld7\" (UniqueName: \"kubernetes.io/projected/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-kube-api-access-xpld7\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516501 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06587788-5cdc-4c3a-be0b-21dc0414aee8-trusted-ca\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdck\" (UniqueName: \"kubernetes.io/projected/dbaf0501-e6b9-46f1-a830-53b1a41dd7ab-kube-api-access-9xdck\") pod \"ingress-canary-l6mrb\" (UID: \"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab\") " pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-klusterlet-config\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-ca\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.516644 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c27888c0-7a26-4595-8baa-8ea243216c6b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-856b685fd-cqhkh\" (UID: \"c27888c0-7a26-4595-8baa-8ea243216c6b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06587788-5cdc-4c3a-be0b-21dc0414aee8-registry-certificates\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-tmp\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p89kz\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-kube-api-access-p89kz\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-hub\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516849 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-image-registry-private-configuration\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516868 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-registry-tls\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.517125 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.516882 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d7c79a1c-6dde-4e5a-8963-8623d9459941-installation-pull-secrets\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.517523 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.517236 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/06587788-5cdc-4c3a-be0b-21dc0414aee8-ca-trust-extracted\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.518725 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.518337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/224dea5b-9272-45de-a02b-2a6c556e445d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.519482 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.519454 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:31.519589 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.519559 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/06587788-5cdc-4c3a-be0b-21dc0414aee8-registry-certificates\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.519700 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.519677 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/06587788-5cdc-4c3a-be0b-21dc0414aee8-image-registry-private-configuration\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.519765 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.519684 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-kube-api-access-lzss5" (OuterVolumeSpecName: "kube-api-access-lzss5") pod "d7c79a1c-6dde-4e5a-8963-8623d9459941" (UID: "d7c79a1c-6dde-4e5a-8963-8623d9459941"). InnerVolumeSpecName "kube-api-access-lzss5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:31.519765 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.519715 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/06587788-5cdc-4c3a-be0b-21dc0414aee8-installation-pull-secrets\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.519994 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.519970 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-hub\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.520245 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.520224 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-tmp\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.520435 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.520226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-registry-tls\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.520597 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.520547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/06587788-5cdc-4c3a-be0b-21dc0414aee8-trusted-ca\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.521937 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.521922 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-ca\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.522744 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.522416 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c27888c0-7a26-4595-8baa-8ea243216c6b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-856b685fd-cqhkh\" (UID: \"c27888c0-7a26-4595-8baa-8ea243216c6b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.522744 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.522606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.522983 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.522959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-klusterlet-config\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.523129 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.523111 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/224dea5b-9272-45de-a02b-2a6c556e445d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.523451 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.523426 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dbaf0501-e6b9-46f1-a830-53b1a41dd7ab-cert\") pod \"ingress-canary-l6mrb\" (UID: \"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab\") " pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.524463 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.524436 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpld7\" (UniqueName: \"kubernetes.io/projected/4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f-kube-api-access-xpld7\") pod \"klusterlet-addon-workmgr-9bf6b68-4687k\" (UID: \"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.524704 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.524659 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wcdt\" (UniqueName: \"kubernetes.io/projected/224dea5b-9272-45de-a02b-2a6c556e445d-kube-api-access-5wcdt\") pod \"cluster-proxy-proxy-agent-5c64945767-hbcp8\" (UID: \"224dea5b-9272-45de-a02b-2a6c556e445d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.525173 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.525136 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-bound-sa-token\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.525650 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.525629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhw7j\" (UniqueName: \"kubernetes.io/projected/c27888c0-7a26-4595-8baa-8ea243216c6b-kube-api-access-nhw7j\") pod \"managed-serviceaccount-addon-agent-856b685fd-cqhkh\" (UID: \"c27888c0-7a26-4595-8baa-8ea243216c6b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.525872 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.525855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdck\" (UniqueName: \"kubernetes.io/projected/dbaf0501-e6b9-46f1-a830-53b1a41dd7ab-kube-api-access-9xdck\") pod \"ingress-canary-l6mrb\" (UID: \"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab\") " pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.526533 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.526518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p89kz\" (UniqueName: \"kubernetes.io/projected/06587788-5cdc-4c3a-be0b-21dc0414aee8-kube-api-access-p89kz\") pod \"image-registry-fd8dbc94d-4df67\" (UID: \"06587788-5cdc-4c3a-be0b-21dc0414aee8\") " pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.532321 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.532306 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-txb7z" Jul 28 00:54:31.552745 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.552677 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l6mrb" Jul 28 00:54:31.577581 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.577549 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" Jul 28 00:54:31.581311 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.581222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:31.603786 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.603574 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" Jul 28 00:54:31.622524 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.622441 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzss5\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-kube-api-access-lzss5\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.622524 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.622474 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d7c79a1c-6dde-4e5a-8963-8623d9459941-bound-sa-token\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:54:31.646704 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.646423 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:31.688076 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.687950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nk2n7"] Jul 28 00:54:31.697791 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.697763 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-twvxb"] Jul 28 00:54:31.707626 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:31.705691 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfa93cf_d7aa_4ba6_a01d_c60611cdfb34.slice/crio-46f4cb87c30bfcbe3d5e93873ff6003f2819d6fc0dd8d8f5d0a34e7403a75ef3 WatchSource:0}: Error finding container 46f4cb87c30bfcbe3d5e93873ff6003f2819d6fc0dd8d8f5d0a34e7403a75ef3: Status 404 returned error can't find the container with id 46f4cb87c30bfcbe3d5e93873ff6003f2819d6fc0dd8d8f5d0a34e7403a75ef3 Jul 28 00:54:31.791449 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:31.791124 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd775c74_88f4_4574_9c7b_eae74a9fb32a.slice/crio-f5143414e211fa23554b5da874e68fcf177d7d9249107e8057b5afd38adb69a5 WatchSource:0}: Error finding container f5143414e211fa23554b5da874e68fcf177d7d9249107e8057b5afd38adb69a5: Status 404 returned error can't find the container with id f5143414e211fa23554b5da874e68fcf177d7d9249107e8057b5afd38adb69a5 Jul 28 00:54:31.794734 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:31.793809 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbaf0501_e6b9_46f1_a830_53b1a41dd7ab.slice/crio-cd092220a65846b31354dfe0b2cc2bc897a3f98199fe9cda25f2f2a37ba32452 WatchSource:0}: Error finding container cd092220a65846b31354dfe0b2cc2bc897a3f98199fe9cda25f2f2a37ba32452: Status 404 returned error can't find the container with id cd092220a65846b31354dfe0b2cc2bc897a3f98199fe9cda25f2f2a37ba32452 Jul 28 00:54:31.794734 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.794292 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-txb7z"] Jul 28 00:54:31.795215 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.794602 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l6mrb"] Jul 28 00:54:31.825688 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.825658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh"] Jul 28 00:54:31.827629 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.827602 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k"] Jul 28 00:54:31.831044 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:31.831022 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc27888c0_7a26_4595_8baa_8ea243216c6b.slice/crio-2e91f9fd2f37ac363ee8d57931c51c3740fd027b7f2971815bb9cb94a166b788 WatchSource:0}: Error finding container 2e91f9fd2f37ac363ee8d57931c51c3740fd027b7f2971815bb9cb94a166b788: Status 404 returned error can't find the container with id 2e91f9fd2f37ac363ee8d57931c51c3740fd027b7f2971815bb9cb94a166b788 Jul 28 00:54:31.832630 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:31.832609 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0f4a5d_e36c_48c1_84a6_c0f50cd75d9f.slice/crio-9287d2ede81f14f20bd46f318107d59192b6d68885640105dc0b085534576d1d WatchSource:0}: Error finding container 9287d2ede81f14f20bd46f318107d59192b6d68885640105dc0b085534576d1d: Status 404 returned error can't find the container with id 9287d2ede81f14f20bd46f318107d59192b6d68885640105dc0b085534576d1d Jul 28 00:54:31.845459 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.845433 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8"] Jul 28 00:54:31.848277 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:31.848226 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod224dea5b_9272_45de_a02b_2a6c556e445d.slice/crio-55c5084216843c5fa207e7f8f381d38c4619351d43eea1410a361fc0e655fd29 WatchSource:0}: Error finding container 55c5084216843c5fa207e7f8f381d38c4619351d43eea1410a361fc0e655fd29: Status 404 returned error can't find the container with id 55c5084216843c5fa207e7f8f381d38c4619351d43eea1410a361fc0e655fd29 Jul 28 00:54:31.861698 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:31.861652 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fd8dbc94d-4df67"] Jul 28 00:54:31.864597 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:31.864577 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06587788_5cdc_4c3a_be0b_21dc0414aee8.slice/crio-173406f21e7d6d5cae7e4a7332eb8cd5ee923ca9a0d73b8e142d0a92ed0a306e WatchSource:0}: Error finding container 173406f21e7d6d5cae7e4a7332eb8cd5ee923ca9a0d73b8e142d0a92ed0a306e: Status 404 returned error can't find the container with id 173406f21e7d6d5cae7e4a7332eb8cd5ee923ca9a0d73b8e142d0a92ed0a306e Jul 28 00:54:32.193207 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.193131 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6mrb" event={"ID":"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab","Type":"ContainerStarted","Data":"cd092220a65846b31354dfe0b2cc2bc897a3f98199fe9cda25f2f2a37ba32452"} Jul 28 00:54:32.194350 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.194326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-txb7z" event={"ID":"fd775c74-88f4-4574-9c7b-eae74a9fb32a","Type":"ContainerStarted","Data":"f5143414e211fa23554b5da874e68fcf177d7d9249107e8057b5afd38adb69a5"} Jul 28 00:54:32.195458 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.195433 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2n7" event={"ID":"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6","Type":"ContainerStarted","Data":"2ca638722f42f5dc5a17984d47f692338dc8b67de94a96bf82d0748ed8216959"} Jul 28 00:54:32.197019 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.196995 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" event={"ID":"06587788-5cdc-4c3a-be0b-21dc0414aee8","Type":"ContainerStarted","Data":"b65f1ac26884677c53a47b127202cb57ef03749752fb5f9dd4ef929938341db8"} Jul 28 00:54:32.197139 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.197039 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" event={"ID":"06587788-5cdc-4c3a-be0b-21dc0414aee8","Type":"ContainerStarted","Data":"173406f21e7d6d5cae7e4a7332eb8cd5ee923ca9a0d73b8e142d0a92ed0a306e"} Jul 28 00:54:32.197139 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.197117 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:32.198188 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.198169 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" event={"ID":"c27888c0-7a26-4595-8baa-8ea243216c6b","Type":"ContainerStarted","Data":"2e91f9fd2f37ac363ee8d57931c51c3740fd027b7f2971815bb9cb94a166b788"} Jul 28 00:54:32.199251 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.199231 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twvxb" event={"ID":"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34","Type":"ContainerStarted","Data":"46f4cb87c30bfcbe3d5e93873ff6003f2819d6fc0dd8d8f5d0a34e7403a75ef3"} Jul 28 00:54:32.200461 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.200441 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" event={"ID":"224dea5b-9272-45de-a02b-2a6c556e445d","Type":"ContainerStarted","Data":"55c5084216843c5fa207e7f8f381d38c4619351d43eea1410a361fc0e655fd29"} Jul 28 00:54:32.201503 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.201484 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" event={"ID":"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f","Type":"ContainerStarted","Data":"9287d2ede81f14f20bd46f318107d59192b6d68885640105dc0b085534576d1d"} Jul 28 00:54:32.201584 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.201514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-775fbd7745-kgh54" Jul 28 00:54:32.214444 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.214403 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" podStartSLOduration=1.214361116 podStartE2EDuration="1.214361116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:32.213655311 +0000 UTC m=+52.957162813" watchObservedRunningTime="2026-07-28 00:54:32.214361116 +0000 UTC m=+52.957868599" Jul 28 00:54:32.241626 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.241573 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-775fbd7745-kgh54"] Jul 28 00:54:32.246457 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.246419 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-775fbd7745-kgh54"] Jul 28 00:54:32.748356 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.748321 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:32.848639 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:32.848513 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:33.793995 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:33.793784 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c79a1c-6dde-4e5a-8963-8623d9459941" path="/var/lib/kubelet/pods/d7c79a1c-6dde-4e5a-8963-8623d9459941/volumes" Jul 28 00:54:36.728349 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:36.728313 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:36.824390 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:36.824328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:36.924195 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:36.924165 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:37.385888 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:37.385858 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:37.454896 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:37.454870 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:37.507067 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:37.507022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:37.600720 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:37.600616 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:40.152345 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:40.152315 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c28hc" Jul 28 00:54:40.298598 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:40.298559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:40.399988 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:40.399915 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:40.500423 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:40.500311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:41.487958 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:41.487907 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:41.585857 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:41.585741 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:41.912462 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:41.912412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:44.108559 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.108523 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:44.170396 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.170346 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:44.450773 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.450683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:44.453123 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.453089 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 00:54:44.463545 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.463520 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321b307e-eb62-4af8-beda-c5bd51c8a9ae-original-pull-secret\") pod \"global-pull-secret-syncer-f6w4c\" (UID: \"321b307e-eb62-4af8-beda-c5bd51c8a9ae\") " pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:44.551337 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.551294 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:44.553668 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.553644 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 00:54:44.564935 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.564906 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa58ed3c-0998-420d-83f1-82a206ed6cd5-metrics-certs\") pod \"network-metrics-daemon-8b67z\" (UID: \"aa58ed3c-0998-420d-83f1-82a206ed6cd5\") " pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:44.597033 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.597009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f6w4c" Jul 28 00:54:44.652604 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.652569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:44.655013 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.654988 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 00:54:44.665506 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.665489 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 00:54:44.677220 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.677196 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lvx\" (UniqueName: \"kubernetes.io/projected/4a864ab0-5e7a-4aa3-bc10-64c02367f7d1-kube-api-access-v2lvx\") pod \"network-check-target-5fbnk\" (UID: \"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1\") " pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:44.801020 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.800986 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cknlx\"" Jul 28 00:54:44.806169 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.806145 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wktlp\"" Jul 28 00:54:44.808892 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.808870 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:54:44.814650 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:44.814629 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8b67z" Jul 28 00:54:45.278744 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:45.278705 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:45.510106 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:45.510075 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:46.117006 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:46.116975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:48.629376 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.629344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:48.849848 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.849809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:48.924096 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.924024 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9lwgx"] Jul 28 00:54:48.948207 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.948160 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:48.955376 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.955342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.957876 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.957853 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 00:54:48.958109 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.958088 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 00:54:48.958189 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.958088 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 00:54:48.958189 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.958120 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 00:54:48.958350 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.958328 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 00:54:48.958788 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.958770 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 00:54:48.958923 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.958901 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9zh8t\"" Jul 28 00:54:48.984704 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-wtmp\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984814 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984814 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984734 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-root\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984814 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0aad6-1625-462c-a7b7-6ad9588599f6-metrics-client-ca\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984978 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984978 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvkf\" (UniqueName: \"kubernetes.io/projected/b5b0aad6-1625-462c-a7b7-6ad9588599f6-kube-api-access-scvkf\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984978 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984912 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-tls\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984978 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-sys\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:48.984978 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:48.984965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-textfile\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.009988 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.009960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.085948 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.085918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-textfile\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086123 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.085956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-wtmp\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086123 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.085983 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086123 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-root\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086130 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-wtmp\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0aad6-1625-462c-a7b7-6ad9588599f6-metrics-client-ca\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086218 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-root\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvkf\" (UniqueName: \"kubernetes.io/projected/b5b0aad6-1625-462c-a7b7-6ad9588599f6-kube-api-access-scvkf\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-textfile\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086295 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-tls\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086640 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:49.086393 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 00:54:49.086640 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:54:49.086454 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-tls podName:b5b0aad6-1625-462c-a7b7-6ad9588599f6 nodeName:}" failed. No retries permitted until 2026-07-28 00:54:49.586432981 +0000 UTC m=+70.329940483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-tls") pod "node-exporter-9lwgx" (UID: "b5b0aad6-1625-462c-a7b7-6ad9588599f6") : secret "node-exporter-tls" not found Jul 28 00:54:49.086640 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-sys\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086798 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5b0aad6-1625-462c-a7b7-6ad9588599f6-sys\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086798 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b0aad6-1625-462c-a7b7-6ad9588599f6-metrics-client-ca\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.086989 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.086972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.089026 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.089005 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.104574 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.104550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvkf\" (UniqueName: \"kubernetes.io/projected/b5b0aad6-1625-462c-a7b7-6ad9588599f6-kube-api-access-scvkf\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.156254 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.156225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.248759 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.248647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.357114 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.357082 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.457220 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.457184 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.558569 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.558521 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.591229 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.591185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-tls\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.593691 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.593668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5b0aad6-1625-462c-a7b7-6ad9588599f6-node-exporter-tls\") pod \"node-exporter-9lwgx\" (UID: \"b5b0aad6-1625-462c-a7b7-6ad9588599f6\") " pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.727962 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.727929 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.831013 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.830920 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.865829 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.865802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lwgx" Jul 28 00:54:49.934127 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.934085 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:49.993555 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:49.993502 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:50.648026 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:50.647977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:50.748848 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:50.748801 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:50.801263 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:50.801210 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:50.877843 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:50.877810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:51.650861 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:51.650831 2580 patch_prober.go:28] interesting pod/image-registry-fd8dbc94d-4df67 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 28 00:54:51.650984 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:51.650882 2580 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" podUID="06587788-5cdc-4c3a-be0b-21dc0414aee8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 00:54:51.836152 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:51.836111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:51.936988 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:51.936912 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:52.015723 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:52.015697 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b0aad6_1625_462c_a7b7_6ad9588599f6.slice/crio-50f2f057fd2b7082ed85c5b0c73b36a173c5a17019ad6d30354abb1be0381585 WatchSource:0}: Error finding container 50f2f057fd2b7082ed85c5b0c73b36a173c5a17019ad6d30354abb1be0381585: Status 404 returned error can't find the container with id 50f2f057fd2b7082ed85c5b0c73b36a173c5a17019ad6d30354abb1be0381585 Jul 28 00:54:52.038934 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.038895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:52.197120 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.196260 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8b67z"] Jul 28 00:54:52.223441 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.223065 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f6w4c"] Jul 28 00:54:52.224920 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.224378 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5fbnk"] Jul 28 00:54:52.225402 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:54:52.225337 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321b307e_eb62_4af8_beda_c5bd51c8a9ae.slice/crio-d00c43508bfc222e6aa0cfe590254a8c6ce426a6c925d9349d38fd72b729ebf8 WatchSource:0}: Error finding container d00c43508bfc222e6aa0cfe590254a8c6ce426a6c925d9349d38fd72b729ebf8: Status 404 returned error can't find the container with id d00c43508bfc222e6aa0cfe590254a8c6ce426a6c925d9349d38fd72b729ebf8 Jul 28 00:54:52.294577 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.294542 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l6mrb" event={"ID":"dbaf0501-e6b9-46f1-a830-53b1a41dd7ab","Type":"ContainerStarted","Data":"cb35d47639124f0868c721fd00e08cb0cf5d1c15055facc96ee412d8922c5cd4"} Jul 28 00:54:52.311194 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.309750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-txb7z" event={"ID":"fd775c74-88f4-4574-9c7b-eae74a9fb32a","Type":"ContainerStarted","Data":"2eb4104d710cf456140475ad40550e6d20242ff60c9ed8e7a2d8faadfeeb17d5"} Jul 28 00:54:52.311929 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.311831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-txb7z" Jul 28 00:54:52.318780 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.313148 2580 patch_prober.go:28] interesting pod/downloads-7c97dc4555-txb7z container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused" start-of-body= Jul 28 00:54:52.318780 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.313199 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-txb7z" podUID="fd775c74-88f4-4574-9c7b-eae74a9fb32a" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.9:8080/\": dial tcp 10.133.0.9:8080: connect: connection refused" Jul 28 00:54:52.318780 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.316589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2n7" event={"ID":"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6","Type":"ContainerStarted","Data":"0e5686821a6d6866e7c3c05fc42510e0e45be6eabaeb2fa071d9a136fd9a53ae"} Jul 28 00:54:52.318780 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.318441 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b67z" event={"ID":"aa58ed3c-0998-420d-83f1-82a206ed6cd5","Type":"ContainerStarted","Data":"c929e29ff8dd6624fb93599ea9d8f21b87ea6570704a2d68b8187decf98df6a5"} Jul 28 00:54:52.324773 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.324748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwgx" event={"ID":"b5b0aad6-1625-462c-a7b7-6ad9588599f6","Type":"ContainerStarted","Data":"50f2f057fd2b7082ed85c5b0c73b36a173c5a17019ad6d30354abb1be0381585"} Jul 28 00:54:52.330689 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.330144 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" event={"ID":"c27888c0-7a26-4595-8baa-8ea243216c6b","Type":"ContainerStarted","Data":"d599cb697e5357795cf6f671f7ba37f461c9ae897f4916c98cb478b1818d54a7"} Jul 28 00:54:52.335324 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.335302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twvxb" event={"ID":"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34","Type":"ContainerStarted","Data":"6c28a06ab0dda904eed70e7f1d1023b31d9864713d9473e34dac4dd584c83295"} Jul 28 00:54:52.337494 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.336744 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-txb7z" podStartSLOduration=1.042919999 podStartE2EDuration="21.336727687s" totalImagesPullingTime="20.293807688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:52.335945501 +0000 UTC m=+73.079453008" watchObservedRunningTime="2026-07-28 00:54:52.336727687 +0000 UTC m=+73.080235189" Jul 28 00:54:52.337494 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.337177 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l6mrb" podStartSLOduration=1.5019493320000001 podStartE2EDuration="21.337166546s" totalImagesPullingTime="19.835217214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:52.311994365 +0000 UTC m=+73.055501958" watchObservedRunningTime="2026-07-28 00:54:52.337166546 +0000 UTC m=+73.080674055" Jul 28 00:54:52.338722 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.338655 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5fbnk" event={"ID":"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1","Type":"ContainerStarted","Data":"fb6f3b53cb0219cbae96492f0523723c1f54fe493b56c56b9cb28fabf3c09715"} Jul 28 00:54:52.341229 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.341173 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6w4c" event={"ID":"321b307e-eb62-4af8-beda-c5bd51c8a9ae","Type":"ContainerStarted","Data":"d00c43508bfc222e6aa0cfe590254a8c6ce426a6c925d9349d38fd72b729ebf8"} Jul 28 00:54:52.344404 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.344340 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" event={"ID":"224dea5b-9272-45de-a02b-2a6c556e445d","Type":"ContainerStarted","Data":"aea819f2239e530b9bd16e018791cea76215f675ae302040baded0fb0884409c"} Jul 28 00:54:52.346376 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.346213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" event={"ID":"4e0f4a5d-e36c-48c1-84a6-c0f50cd75d9f","Type":"ContainerStarted","Data":"cf3d3290859cfe307ba50836fb60b69ef83cdb1cdbf9384c60b829496cffdd23"} Jul 28 00:54:52.346814 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.346797 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:52.348176 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.348162 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" Jul 28 00:54:52.351962 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.351920 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-856b685fd-cqhkh" podStartSLOduration=8.479347018 podStartE2EDuration="21.35190171s" totalImagesPullingTime="12.872554692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:52.350482009 +0000 UTC m=+73.093989515" watchObservedRunningTime="2026-07-28 00:54:52.35190171 +0000 UTC m=+73.095409205" Jul 28 00:54:52.368898 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.368844 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-9bf6b68-4687k" podStartSLOduration=1.230228117 podStartE2EDuration="21.368823568s" totalImagesPullingTime="20.138595451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:52.367767195 +0000 UTC m=+73.111274700" watchObservedRunningTime="2026-07-28 00:54:52.368823568 +0000 UTC m=+73.112331075" Jul 28 00:54:52.381552 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:52.381478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:53.221389 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.221153 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fd8dbc94d-4df67" Jul 28 00:54:53.286259 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.286192 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:53.340779 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.340745 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:53.364338 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.364295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-twvxb" event={"ID":"edfa93cf-d7aa-4ba6-a01d-c60611cdfb34","Type":"ContainerStarted","Data":"308cf783f0406cce7bb14a88493fe7f16192de1ee42f541afc1e3d6959e817d3"} Jul 28 00:54:53.364545 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.364409 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-twvxb" Jul 28 00:54:53.369420 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.369392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" event={"ID":"224dea5b-9272-45de-a02b-2a6c556e445d","Type":"ContainerStarted","Data":"204a095a73ed0e64953b69e3a08bfd22b221318bbb1124bd17d59c6602542cf2"} Jul 28 00:54:53.369555 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.369435 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" event={"ID":"224dea5b-9272-45de-a02b-2a6c556e445d","Type":"ContainerStarted","Data":"7ab529c48f0d31f5442bb496600f4bf9ea724e226f2fe6adc7d22bdf20af3385"} Jul 28 00:54:53.380217 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.380166 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-twvxb" podStartSLOduration=2.455183521 podStartE2EDuration="22.380149342s" totalImagesPullingTime="19.924965821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:53.379404773 +0000 UTC m=+74.122912278" watchObservedRunningTime="2026-07-28 00:54:53.380149342 +0000 UTC m=+74.123656847" Jul 28 00:54:53.387828 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.387808 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-txb7z" Jul 28 00:54:53.398043 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.397716 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5c64945767-hbcp8" podStartSLOduration=2.275300759 podStartE2EDuration="22.397694631s" totalImagesPullingTime="20.122393872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:53.39681536 +0000 UTC m=+74.140322864" watchObservedRunningTime="2026-07-28 00:54:53.397694631 +0000 UTC m=+74.141202137" Jul 28 00:54:53.402500 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.402408 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:53.664103 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.664003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:53.736035 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:53.735902 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:54.358714 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:54.358564 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:54.679627 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:54.679498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:54.750997 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:54.750964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:55.144686 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.144310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:55.286115 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.286057 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:55.384387 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.383777 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b67z" event={"ID":"aa58ed3c-0998-420d-83f1-82a206ed6cd5","Type":"ContainerStarted","Data":"df290f4288bcf75fae35901ca1fa3e39264b2f51e365bb6d40c6969041379aa9"} Jul 28 00:54:55.384387 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.383844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8b67z" event={"ID":"aa58ed3c-0998-420d-83f1-82a206ed6cd5","Type":"ContainerStarted","Data":"e0b1698414bd1e3057e87bdb789e64898f4f73ef0fc623fffbba8635377e4316"} Jul 28 00:54:55.388410 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.387207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:55.388410 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.387606 2580 generic.go:356] "Generic (PLEG): container finished" podID="b5b0aad6-1625-462c-a7b7-6ad9588599f6" containerID="a0dacd363bf7966ae3cec3b0429b7931b17a7ed727b025280d9cbcfbcfee5c2b" exitCode=0 Jul 28 00:54:55.389547 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.389455 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwgx" event={"ID":"b5b0aad6-1625-462c-a7b7-6ad9588599f6","Type":"ContainerDied","Data":"a0dacd363bf7966ae3cec3b0429b7931b17a7ed727b025280d9cbcfbcfee5c2b"} Jul 28 00:54:55.403772 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.402635 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8b67z" podStartSLOduration=73.685942784 podStartE2EDuration="1m16.402618783s" totalImagesPullingTime="2.716675999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:55.402064946 +0000 UTC m=+76.145572451" watchObservedRunningTime="2026-07-28 00:54:55.402618783 +0000 UTC m=+76.146126288" Jul 28 00:54:55.489032 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.488749 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:55.547389 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:55.547250 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:56.399681 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:56.399645 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwgx" event={"ID":"b5b0aad6-1625-462c-a7b7-6ad9588599f6","Type":"ContainerStarted","Data":"7700d7eb4e06675309fb4d98bd8ed1cf76b1a3b94435a1e883f305f9138afe4d"} Jul 28 00:54:56.400150 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:56.399701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lwgx" event={"ID":"b5b0aad6-1625-462c-a7b7-6ad9588599f6","Type":"ContainerStarted","Data":"de90c5dbfbaf4653620face2ca297510639e9a12133ddca65308e0545d05db82"} Jul 28 00:54:56.424395 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:56.424289 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9lwgx" podStartSLOduration=5.5924513860000005 podStartE2EDuration="8.424267668s" totalImagesPullingTime="2.692258606s" totalInitContainerRuntime="139.557676ms" isStatefulPod=true podCreationTimestamp="2026-07-28 00:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:56.421494539 +0000 UTC m=+77.165002044" watchObservedRunningTime="2026-07-28 00:54:56.424267668 +0000 UTC m=+77.167775173" Jul 28 00:54:57.182343 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:57.182308 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:54:57.283102 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:54:57.283026 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:00.282900 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:00.282856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:00.329884 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:00.329850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:03.105765 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.105736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:03.203869 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.203790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:03.379389 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.379351 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-twvxb" Jul 28 00:55:03.429435 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.429408 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5fbnk" event={"ID":"4a864ab0-5e7a-4aa3-bc10-64c02367f7d1","Type":"ContainerStarted","Data":"b70e0f46539642d93138bf2af845f777b2a97675f1be837f4dfbe968be19b3d7"} Jul 28 00:55:03.429600 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.429493 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:55:03.431002 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.430966 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f6w4c" event={"ID":"321b307e-eb62-4af8-beda-c5bd51c8a9ae","Type":"ContainerStarted","Data":"e95e811105c81279e361c6d011fc1fc1fbb83135253f31414b3fa8bfda35cc9f"} Jul 28 00:55:03.444877 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.444820 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5fbnk" podStartSLOduration=73.689748194 podStartE2EDuration="1m24.444802038s" totalImagesPullingTime="10.755053844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:03.443146906 +0000 UTC m=+84.186654410" watchObservedRunningTime="2026-07-28 00:55:03.444802038 +0000 UTC m=+84.188309543" Jul 28 00:55:03.456414 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.456308 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-f6w4c" podStartSLOduration=73.695010134 podStartE2EDuration="1m24.456289566s" totalImagesPullingTime="10.761279432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:03.456057375 +0000 UTC m=+84.199564879" watchObservedRunningTime="2026-07-28 00:55:03.456289566 +0000 UTC m=+84.199797070" Jul 28 00:55:03.490492 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.490467 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:03.591659 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:03.591627 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:04.436753 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:04.436705 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2n7" event={"ID":"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6","Type":"ContainerStarted","Data":"feb5629759434f1bfb9d7831e885241d9f958b6bab3e1a5e56deef10c02898b3"} Jul 28 00:55:04.939726 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:04.939694 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:04.996911 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:04.996871 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:08.452261 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:08.452227 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nk2n7" event={"ID":"9e5a3055-47c7-4ccc-9907-a7c96db9b9a6","Type":"ContainerStarted","Data":"2f95d9a0424ee84e63bf5596d3f3bcbb6eaf3ff7d9f25aed7ff907f49ff1abe6"} Jul 28 00:55:08.470569 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:08.470515 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nk2n7" podStartSLOduration=1.923464788 podStartE2EDuration="37.470496192s" totalImagesPullingTime="35.547031404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:08.468897036 +0000 UTC m=+89.212404540" watchObservedRunningTime="2026-07-28 00:55:08.470496192 +0000 UTC m=+89.214003698" Jul 28 00:55:09.368135 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:09.368096 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:09.416452 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:09.416420 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:12.354938 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:12.354906 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:12.404051 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:12.404012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:15.190320 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:15.190289 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:15.270935 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:15.270908 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:16.404048 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:16.404017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:27.361854 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:27.361820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:27.411382 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:27.411340 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:29.235463 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:29.235432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:29.291225 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:29.291197 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:55:34.439809 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:34.439781 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5fbnk" Jul 28 00:55:46.113992 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:55:46.113960 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:04.982611 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:04.982570 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:05.031688 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:05.031655 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:16.828090 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:16.828016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:16.886778 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:16.886748 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:20.494049 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:20.494012 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:20.550078 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:20.550038 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:20.802320 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:20.802283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:21.414170 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:21.414139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:21.461701 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:21.461671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:27.370000 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:27.369966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:27.467762 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:27.467730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:31.744624 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:31.744545 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:31.796636 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:31.796605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:47.903845 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:47.903799 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:49.374022 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:49.373984 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:56:49.474701 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:56:49.474668 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:10.018174 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:10.018139 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:10.118980 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:10.118951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:16.183441 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:16.183383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:17.555494 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:17.555461 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:20.001752 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:20.001721 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:20.100732 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:20.100702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:21.298944 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:21.298910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:21.398280 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:21.398253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:22.329889 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:22.329856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:22.431994 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:22.431964 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:24.355395 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:24.355341 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:24.708138 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:24.708042 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:26.986027 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:26.985992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:27.052886 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:27.052852 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:29.323723 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:29.323691 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:29.432685 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:29.432656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:32.918408 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:32.918356 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:33.026493 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:33.026460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:33.119688 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:33.119660 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:33.223590 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:33.223522 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:39.141164 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:39.141120 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:39.244708 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:39.244673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:41.337955 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:41.337922 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:42.653979 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:42.653945 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:55.652035 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:55.652005 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:55.755359 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:55.755326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:56.243500 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.243466 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:56.494063 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.493992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:56.599872 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.599838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:56.651131 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.651106 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p"] Jul 28 00:57:56.654159 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.654140 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.656696 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.656674 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 00:57:56.656788 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.656727 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cm2gp\"" Jul 28 00:57:56.657193 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.657177 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 00:57:56.657423 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.657409 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 00:57:56.657496 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.657489 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 00:57:56.672965 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.672945 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p"] Jul 28 00:57:56.787846 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.787821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:56.809020 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.808994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b92f522-ccd5-47d2-b969-c0615b31a545-tmp\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.809131 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.809028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/4b92f522-ccd5-47d2-b969-c0615b31a545-kube-api-access-n2vwg\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.809131 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.809058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b92f522-ccd5-47d2-b969-c0615b31a545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.809202 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.809122 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b92f522-ccd5-47d2-b969-c0615b31a545-manifests\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.809202 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.809167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b92f522-ccd5-47d2-b969-c0615b31a545-webhook-cert\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.888832 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.888806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:56.910450 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.910427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b92f522-ccd5-47d2-b969-c0615b31a545-tmp\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.910539 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.910458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/4b92f522-ccd5-47d2-b969-c0615b31a545-kube-api-access-n2vwg\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.910539 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.910494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b92f522-ccd5-47d2-b969-c0615b31a545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.910539 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.910525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b92f522-ccd5-47d2-b969-c0615b31a545-manifests\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.910694 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.910552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b92f522-ccd5-47d2-b969-c0615b31a545-webhook-cert\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.910881 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.910862 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b92f522-ccd5-47d2-b969-c0615b31a545-manifests\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.913233 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.913203 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b92f522-ccd5-47d2-b969-c0615b31a545-tmp\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.913355 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.913337 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b92f522-ccd5-47d2-b969-c0615b31a545-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.913432 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.913339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b92f522-ccd5-47d2-b969-c0615b31a545-webhook-cert\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.925969 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.925948 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vwg\" (UniqueName: \"kubernetes.io/projected/4b92f522-ccd5-47d2-b969-c0615b31a545-kube-api-access-n2vwg\") pod \"opendatahub-operator-controller-manager-6549c9954f-cpb7p\" (UID: \"4b92f522-ccd5-47d2-b969-c0615b31a545\") " pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.963967 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.963944 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:57:56.997597 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:56.997569 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:57.091522 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:57.091492 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p"] Jul 28 00:57:57.094175 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:57.094143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:57.173623 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:57.173591 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:57.278709 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:57.278671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:57.368852 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:57.368770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:57:58.004958 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:57:58.004916 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" event={"ID":"4b92f522-ccd5-47d2-b969-c0615b31a545","Type":"ContainerStarted","Data":"65980beef5c6fccfe70f3e3ccea67cb24a1263e52a2f7e8dc1e2c1c2425eec67"} Jul 28 00:58:01.017740 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:01.017706 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" event={"ID":"4b92f522-ccd5-47d2-b969-c0615b31a545","Type":"ContainerStarted","Data":"903d2cecaa5f4369a112c6d9e38a1cbdd2151ddaca11116630b2641774b3b53d"} Jul 28 00:58:02.022986 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:02.022952 2580 generic.go:356] "Generic (PLEG): container finished" podID="4b92f522-ccd5-47d2-b969-c0615b31a545" containerID="903d2cecaa5f4369a112c6d9e38a1cbdd2151ddaca11116630b2641774b3b53d" exitCode=0 Jul 28 00:58:02.023400 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:02.022997 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" event={"ID":"4b92f522-ccd5-47d2-b969-c0615b31a545","Type":"ContainerDied","Data":"903d2cecaa5f4369a112c6d9e38a1cbdd2151ddaca11116630b2641774b3b53d"} Jul 28 00:58:03.028772 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:03.028734 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" event={"ID":"4b92f522-ccd5-47d2-b969-c0615b31a545","Type":"ContainerStarted","Data":"19a14b8039d226a07f13e0818f5fd32e85855f1fc32d6392d8efb4ad958446c8"} Jul 28 00:58:03.029177 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:03.028875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:58:03.051969 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:03.051920 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" podStartSLOduration=2.764307594 podStartE2EDuration="7.051903949s" totalImagesPullingTime="3.262531265s" totalInitContainerRuntime="1.02506509s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:57:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:58:03.051172559 +0000 UTC m=+263.794680062" watchObservedRunningTime="2026-07-28 00:58:03.051903949 +0000 UTC m=+263.795411453" Jul 28 00:58:07.600746 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:07.600713 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:07.701490 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:07.701463 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:14.034651 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:14.034618 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6549c9954f-cpb7p" Jul 28 00:58:15.574602 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:15.574568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:15.763509 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:15.763472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:16.335529 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.335495 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:16.452484 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.452458 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h"] Jul 28 00:58:16.460775 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.460755 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.463722 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.463691 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 28 00:58:16.463873 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.463769 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 28 00:58:16.464772 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.464755 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7vwnk\"" Jul 28 00:58:16.464882 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.464772 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 00:58:16.464882 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.464810 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 00:58:16.468296 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.468271 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h"] Jul 28 00:58:16.551110 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.551085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnx89\" (UniqueName: \"kubernetes.io/projected/692be0fe-0a4b-43c8-8ade-5a57b07790a4-kube-api-access-wnx89\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.551257 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.551146 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tmp\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.551257 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.551171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tls-certs\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.652317 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.652250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnx89\" (UniqueName: \"kubernetes.io/projected/692be0fe-0a4b-43c8-8ade-5a57b07790a4-kube-api-access-wnx89\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.652700 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.652320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tmp\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.652700 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.652347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tls-certs\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.652700 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:58:16.652522 2580 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 28 00:58:16.652700 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:58:16.652586 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tls-certs podName:692be0fe-0a4b-43c8-8ade-5a57b07790a4 nodeName:}" failed. No retries permitted until 2026-07-28 00:58:17.152567039 +0000 UTC m=+277.896074526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tls-certs") pod "kube-auth-proxy-fbf9bd945-lp78h" (UID: "692be0fe-0a4b-43c8-8ade-5a57b07790a4") : secret "kube-auth-proxy-tls" not found Jul 28 00:58:16.654512 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.654489 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tmp\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.666172 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.666145 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnx89\" (UniqueName: \"kubernetes.io/projected/692be0fe-0a4b-43c8-8ade-5a57b07790a4-kube-api-access-wnx89\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:16.732434 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.732403 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f2qh4"] Jul 28 00:58:16.734473 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.734457 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:16.737073 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.737050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-p2bgk\"" Jul 28 00:58:16.737170 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.737050 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 28 00:58:16.744069 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.744047 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f2qh4"] Jul 28 00:58:16.753199 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.753180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a317d8a-cd99-4e7a-ad09-869b3978070b-cert\") pod \"odh-model-controller-7998c67dc4-f2qh4\" (UID: \"4a317d8a-cd99-4e7a-ad09-869b3978070b\") " pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:16.753294 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.753230 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjkx\" (UniqueName: \"kubernetes.io/projected/4a317d8a-cd99-4e7a-ad09-869b3978070b-kube-api-access-kpjkx\") pod \"odh-model-controller-7998c67dc4-f2qh4\" (UID: \"4a317d8a-cd99-4e7a-ad09-869b3978070b\") " pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:16.776995 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.776973 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:16.853902 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.853877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a317d8a-cd99-4e7a-ad09-869b3978070b-cert\") pod \"odh-model-controller-7998c67dc4-f2qh4\" (UID: \"4a317d8a-cd99-4e7a-ad09-869b3978070b\") " pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:16.854027 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.853929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjkx\" (UniqueName: \"kubernetes.io/projected/4a317d8a-cd99-4e7a-ad09-869b3978070b-kube-api-access-kpjkx\") pod \"odh-model-controller-7998c67dc4-f2qh4\" (UID: \"4a317d8a-cd99-4e7a-ad09-869b3978070b\") " pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:16.854071 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:58:16.854027 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 28 00:58:16.854104 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:58:16.854089 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a317d8a-cd99-4e7a-ad09-869b3978070b-cert podName:4a317d8a-cd99-4e7a-ad09-869b3978070b nodeName:}" failed. No retries permitted until 2026-07-28 00:58:17.354074117 +0000 UTC m=+278.097581599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4a317d8a-cd99-4e7a-ad09-869b3978070b-cert") pod "odh-model-controller-7998c67dc4-f2qh4" (UID: "4a317d8a-cd99-4e7a-ad09-869b3978070b") : secret "odh-model-controller-webhook-cert" not found Jul 28 00:58:16.864711 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.864693 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjkx\" (UniqueName: \"kubernetes.io/projected/4a317d8a-cd99-4e7a-ad09-869b3978070b-kube-api-access-kpjkx\") pod \"odh-model-controller-7998c67dc4-f2qh4\" (UID: \"4a317d8a-cd99-4e7a-ad09-869b3978070b\") " pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:16.879259 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.879233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:16.984822 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:16.984743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:17.082287 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.082253 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:17.156309 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.156278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tls-certs\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:17.158480 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.158461 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692be0fe-0a4b-43c8-8ade-5a57b07790a4-tls-certs\") pod \"kube-auth-proxy-fbf9bd945-lp78h\" (UID: \"692be0fe-0a4b-43c8-8ade-5a57b07790a4\") " pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:17.357900 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.357861 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a317d8a-cd99-4e7a-ad09-869b3978070b-cert\") pod \"odh-model-controller-7998c67dc4-f2qh4\" (UID: \"4a317d8a-cd99-4e7a-ad09-869b3978070b\") " pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:17.360387 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.360351 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a317d8a-cd99-4e7a-ad09-869b3978070b-cert\") pod \"odh-model-controller-7998c67dc4-f2qh4\" (UID: \"4a317d8a-cd99-4e7a-ad09-869b3978070b\") " pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:17.370214 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.370189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" Jul 28 00:58:17.493180 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.493145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h"] Jul 28 00:58:17.496098 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:58:17.496064 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692be0fe_0a4b_43c8_8ade_5a57b07790a4.slice/crio-857bd477bc471afea0e202d9e0e84209c9d063f3e7f333a6ab09e6ead8d591da WatchSource:0}: Error finding container 857bd477bc471afea0e202d9e0e84209c9d063f3e7f333a6ab09e6ead8d591da: Status 404 returned error can't find the container with id 857bd477bc471afea0e202d9e0e84209c9d063f3e7f333a6ab09e6ead8d591da Jul 28 00:58:17.562165 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.562131 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:17.644010 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.643922 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:17.644775 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.644752 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:17.647246 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.647219 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84"] Jul 28 00:58:17.649407 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.649380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.651560 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.651538 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 00:58:17.651687 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.651576 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xprbh\"" Jul 28 00:58:17.651687 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.651601 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 00:58:17.651687 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.651543 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 00:58:17.651846 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.651833 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 00:58:17.660994 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.660966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.661339 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.661012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.661339 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.661046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.661339 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.661120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlz6l\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-kube-api-access-wlz6l\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.661339 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.661161 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418096c-489a-4949-b392-81238911bd1d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.661339 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.661208 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.661339 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.661239 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d418096c-489a-4949-b392-81238911bd1d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.665678 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.665659 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84"] Jul 28 00:58:17.728184 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.728146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:17.762399 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.762356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.762566 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.762494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.762566 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.762547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlz6l\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-kube-api-access-wlz6l\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.762682 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.762589 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418096c-489a-4949-b392-81238911bd1d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.762682 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.762634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.762682 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.762661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d418096c-489a-4949-b392-81238911bd1d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.762828 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.762707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.763310 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.763280 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d418096c-489a-4949-b392-81238911bd1d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.764909 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.764877 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.764909 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.764888 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.764909 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.764887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418096c-489a-4949-b392-81238911bd1d-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.765083 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.764963 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.770025 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.770001 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.770105 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.770027 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlz6l\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-kube-api-access-wlz6l\") pod \"istiod-openshift-gateway-7b7995d5cd-vkh84\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:17.773926 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.773905 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f2qh4"] Jul 28 00:58:17.776995 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:58:17.776964 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a317d8a_cd99_4e7a_ad09_869b3978070b.slice/crio-e3b288451f5a9e9216fdf7e03f31b65b8b9bd3b6201dc777941e3cb784f9fa81 WatchSource:0}: Error finding container e3b288451f5a9e9216fdf7e03f31b65b8b9bd3b6201dc777941e3cb784f9fa81: Status 404 returned error can't find the container with id e3b288451f5a9e9216fdf7e03f31b65b8b9bd3b6201dc777941e3cb784f9fa81 Jul 28 00:58:17.827304 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.827261 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:17.968338 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:17.968250 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:18.080249 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:18.080215 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" event={"ID":"692be0fe-0a4b-43c8-8ade-5a57b07790a4","Type":"ContainerStarted","Data":"857bd477bc471afea0e202d9e0e84209c9d063f3e7f333a6ab09e6ead8d591da"} Jul 28 00:58:18.081360 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:18.081333 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" event={"ID":"4a317d8a-cd99-4e7a-ad09-869b3978070b","Type":"ContainerStarted","Data":"e3b288451f5a9e9216fdf7e03f31b65b8b9bd3b6201dc777941e3cb784f9fa81"} Jul 28 00:58:18.095207 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:18.095101 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84"] Jul 28 00:58:18.097799 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:58:18.097774 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd418096c_489a_4949_b392_81238911bd1d.slice/crio-14c3ed385e0eeddecee3fc2ec92cee52e4dfcd40eb16163cd1f7f3e680d30868 WatchSource:0}: Error finding container 14c3ed385e0eeddecee3fc2ec92cee52e4dfcd40eb16163cd1f7f3e680d30868: Status 404 returned error can't find the container with id 14c3ed385e0eeddecee3fc2ec92cee52e4dfcd40eb16163cd1f7f3e680d30868 Jul 28 00:58:18.616018 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:18.615971 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:19.089504 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:19.089467 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" event={"ID":"d418096c-489a-4949-b392-81238911bd1d","Type":"ContainerStarted","Data":"14c3ed385e0eeddecee3fc2ec92cee52e4dfcd40eb16163cd1f7f3e680d30868"} Jul 28 00:58:22.554122 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:22.554092 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 28 00:58:22.554570 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:22.554150 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 28 00:58:23.106058 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.106018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" event={"ID":"692be0fe-0a4b-43c8-8ade-5a57b07790a4","Type":"ContainerStarted","Data":"95f230d077455251cd975f5685a8de1954e4f5e8c61eef7898eca7f8aab0e425"} Jul 28 00:58:23.107620 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.107596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" event={"ID":"d418096c-489a-4949-b392-81238911bd1d","Type":"ContainerStarted","Data":"ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641"} Jul 28 00:58:23.107779 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.107747 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:23.109148 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.109118 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" event={"ID":"4a317d8a-cd99-4e7a-ad09-869b3978070b","Type":"ContainerStarted","Data":"dfbce829b04f5671545144a0b789d38557ea7cd3706eb21cc931906bad12b829"} Jul 28 00:58:23.109230 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.109178 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:23.109697 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.109675 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vkh84 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 00:58:23.109775 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.109721 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" podUID="d418096c-489a-4949-b392-81238911bd1d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 00:58:23.131883 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.131840 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-fbf9bd945-lp78h" podStartSLOduration=2.081815812 podStartE2EDuration="7.131828216s" totalImagesPullingTime="5.050012404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:58:23.130927887 +0000 UTC m=+283.874435395" watchObservedRunningTime="2026-07-28 00:58:23.131828216 +0000 UTC m=+283.875335719" Jul 28 00:58:23.162198 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.162136 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" podStartSLOduration=2.379209646 podStartE2EDuration="7.162114454s" totalImagesPullingTime="4.782904808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:58:23.161999586 +0000 UTC m=+283.905507092" watchObservedRunningTime="2026-07-28 00:58:23.162114454 +0000 UTC m=+283.905621964" Jul 28 00:58:23.195892 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.195831 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" podStartSLOduration=1.7416026470000001 podStartE2EDuration="6.195808655s" totalImagesPullingTime="4.454206008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:58:23.194950466 +0000 UTC m=+283.938457970" watchObservedRunningTime="2026-07-28 00:58:23.195808655 +0000 UTC m=+283.939316161" Jul 28 00:58:23.265908 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.265869 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:23.369120 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.368955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:23.806358 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.806324 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x"] Jul 28 00:58:23.809606 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.809589 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.821493 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.821469 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x"] Jul 28 00:58:23.918255 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.918223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.918255 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.918261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.918506 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.918290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.918506 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.918376 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780ed89-46cb-4450-808c-b0ed27f1c3c9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.918506 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.918423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.918506 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.918452 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2whg\" (UniqueName: \"kubernetes.io/projected/9780ed89-46cb-4450-808c-b0ed27f1c3c9-kube-api-access-m2whg\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.918673 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.918510 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:23.942804 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:23.942767 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:24.019699 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.019666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.019862 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.019721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.019862 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.019741 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.019862 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.019771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.019862 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.019805 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780ed89-46cb-4450-808c-b0ed27f1c3c9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.019862 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.019829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.019862 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.019854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2whg\" (UniqueName: \"kubernetes.io/projected/9780ed89-46cb-4450-808c-b0ed27f1c3c9-kube-api-access-m2whg\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.020703 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.020672 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.022203 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.022180 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.022279 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.022223 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.022279 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.022239 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9780ed89-46cb-4450-808c-b0ed27f1c3c9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.022351 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.022275 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.027434 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.027402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:24.028608 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.028584 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9780ed89-46cb-4450-808c-b0ed27f1c3c9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.028980 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.028959 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2whg\" (UniqueName: \"kubernetes.io/projected/9780ed89-46cb-4450-808c-b0ed27f1c3c9-kube-api-access-m2whg\") pod \"istiod-openshift-gateway-5677ccb49b-f9t7x\" (UID: \"9780ed89-46cb-4450-808c-b0ed27f1c3c9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.113848 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.113774 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:24.117931 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.117916 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:24.234081 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.234051 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:24.259040 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.259010 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x"] Jul 28 00:58:24.262482 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:58:24.262455 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9780ed89_46cb_4450_808c_b0ed27f1c3c9.slice/crio-2776a453f9a28f0979c9a6c0aa65ff7763856266eec47b912baf6064b4b3ab2b WatchSource:0}: Error finding container 2776a453f9a28f0979c9a6c0aa65ff7763856266eec47b912baf6064b4b3ab2b: Status 404 returned error can't find the container with id 2776a453f9a28f0979c9a6c0aa65ff7763856266eec47b912baf6064b4b3ab2b Jul 28 00:58:24.265473 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.265443 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 28 00:58:24.265548 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:24.265502 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Jul 28 00:58:25.117780 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:25.117740 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" event={"ID":"9780ed89-46cb-4450-808c-b0ed27f1c3c9","Type":"ContainerStarted","Data":"61b4273624e2b22968d351d9cde60ef4f2778f2c00415fb0c6dbf051b62be974"} Jul 28 00:58:25.118231 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:25.117836 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:25.118231 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:25.117853 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" event={"ID":"9780ed89-46cb-4450-808c-b0ed27f1c3c9","Type":"ContainerStarted","Data":"2776a453f9a28f0979c9a6c0aa65ff7763856266eec47b912baf6064b4b3ab2b"} Jul 28 00:58:25.119549 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:25.119523 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-f9t7x container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 00:58:25.119647 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:25.119568 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" podUID="9780ed89-46cb-4450-808c-b0ed27f1c3c9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 00:58:25.140458 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:25.140410 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" podStartSLOduration=2.140391943 podStartE2EDuration="2.140391943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:58:25.139598458 +0000 UTC m=+285.883105962" watchObservedRunningTime="2026-07-28 00:58:25.140391943 +0000 UTC m=+285.883899449" Jul 28 00:58:26.121813 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.121783 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-f9t7x" Jul 28 00:58:26.195814 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.195786 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84"] Jul 28 00:58:26.195990 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.195969 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" podUID="d418096c-489a-4949-b392-81238911bd1d" containerName="discovery" containerID="cri-o://ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641" gracePeriod=30 Jul 28 00:58:26.455786 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.455759 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:26.509355 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.509326 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:26.540849 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.540809 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-istio-token\") pod \"d418096c-489a-4949-b392-81238911bd1d\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " Jul 28 00:58:26.540849 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.540843 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-csr-dns-cert\") pod \"d418096c-489a-4949-b392-81238911bd1d\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " Jul 28 00:58:26.541072 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.540865 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418096c-489a-4949-b392-81238911bd1d-local-certs\") pod \"d418096c-489a-4949-b392-81238911bd1d\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " Jul 28 00:58:26.541072 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.540880 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-cacerts\") pod \"d418096c-489a-4949-b392-81238911bd1d\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " Jul 28 00:58:26.541072 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.540943 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlz6l\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-kube-api-access-wlz6l\") pod \"d418096c-489a-4949-b392-81238911bd1d\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " Jul 28 00:58:26.541072 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.540964 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d418096c-489a-4949-b392-81238911bd1d-istio-csr-ca-configmap\") pod \"d418096c-489a-4949-b392-81238911bd1d\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " Jul 28 00:58:26.541072 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.541018 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-kubeconfig\") pod \"d418096c-489a-4949-b392-81238911bd1d\" (UID: \"d418096c-489a-4949-b392-81238911bd1d\") " Jul 28 00:58:26.541300 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.541181 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm"] Jul 28 00:58:26.541599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.541577 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d418096c-489a-4949-b392-81238911bd1d" containerName="discovery" Jul 28 00:58:26.541599 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.541600 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d418096c-489a-4949-b392-81238911bd1d" containerName="discovery" Jul 28 00:58:26.541746 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.541674 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d418096c-489a-4949-b392-81238911bd1d" containerName="discovery" Jul 28 00:58:26.541802 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.541771 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d418096c-489a-4949-b392-81238911bd1d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d418096c-489a-4949-b392-81238911bd1d" (UID: "d418096c-489a-4949-b392-81238911bd1d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:58:26.543504 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.543440 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d418096c-489a-4949-b392-81238911bd1d-local-certs" (OuterVolumeSpecName: "local-certs") pod "d418096c-489a-4949-b392-81238911bd1d" (UID: "d418096c-489a-4949-b392-81238911bd1d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:58:26.543678 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.543594 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-cacerts" (OuterVolumeSpecName: "cacerts") pod "d418096c-489a-4949-b392-81238911bd1d" (UID: "d418096c-489a-4949-b392-81238911bd1d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:58:26.544190 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.544136 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d418096c-489a-4949-b392-81238911bd1d" (UID: "d418096c-489a-4949-b392-81238911bd1d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:58:26.544443 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.544418 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d418096c-489a-4949-b392-81238911bd1d" (UID: "d418096c-489a-4949-b392-81238911bd1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:58:26.544570 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.544444 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d418096c-489a-4949-b392-81238911bd1d" (UID: "d418096c-489a-4949-b392-81238911bd1d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:58:26.546579 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.546558 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-kube-api-access-wlz6l" (OuterVolumeSpecName: "kube-api-access-wlz6l") pod "d418096c-489a-4949-b392-81238911bd1d" (UID: "d418096c-489a-4949-b392-81238911bd1d"). InnerVolumeSpecName "kube-api-access-wlz6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:58:26.548354 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.548340 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:26.550856 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.550835 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8s79p\"" Jul 28 00:58:26.550928 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.550860 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 28 00:58:26.554484 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.554466 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm"] Jul 28 00:58:26.642147 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d1a214-2536-4b79-ac51-dc904ee807d7-cert\") pod \"kserve-controller-manager-d6f4b65b8-ttnkm\" (UID: \"72d1a214-2536-4b79-ac51-dc904ee807d7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcq8h\" (UniqueName: \"kubernetes.io/projected/72d1a214-2536-4b79-ac51-dc904ee807d7-kube-api-access-wcq8h\") pod \"kserve-controller-manager-d6f4b65b8-ttnkm\" (UID: \"72d1a214-2536-4b79-ac51-dc904ee807d7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642220 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlz6l\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-kube-api-access-wlz6l\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642234 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d418096c-489a-4949-b392-81238911bd1d-istio-csr-ca-configmap\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642247 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-kubeconfig\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642288 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418096c-489a-4949-b392-81238911bd1d-istio-token\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642309 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-istio-csr-dns-cert\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642318 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418096c-489a-4949-b392-81238911bd1d-local-certs\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:58:26.642336 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.642327 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d418096c-489a-4949-b392-81238911bd1d-cacerts\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:58:26.743141 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.743044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d1a214-2536-4b79-ac51-dc904ee807d7-cert\") pod \"kserve-controller-manager-d6f4b65b8-ttnkm\" (UID: \"72d1a214-2536-4b79-ac51-dc904ee807d7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:26.743141 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.743102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcq8h\" (UniqueName: \"kubernetes.io/projected/72d1a214-2536-4b79-ac51-dc904ee807d7-kube-api-access-wcq8h\") pod \"kserve-controller-manager-d6f4b65b8-ttnkm\" (UID: \"72d1a214-2536-4b79-ac51-dc904ee807d7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:26.745521 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.745502 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72d1a214-2536-4b79-ac51-dc904ee807d7-cert\") pod \"kserve-controller-manager-d6f4b65b8-ttnkm\" (UID: \"72d1a214-2536-4b79-ac51-dc904ee807d7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:26.751544 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.751514 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcq8h\" (UniqueName: \"kubernetes.io/projected/72d1a214-2536-4b79-ac51-dc904ee807d7-kube-api-access-wcq8h\") pod \"kserve-controller-manager-d6f4b65b8-ttnkm\" (UID: \"72d1a214-2536-4b79-ac51-dc904ee807d7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:26.866854 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:26.866819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:27.008777 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.008611 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm"] Jul 28 00:58:27.011307 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:58:27.011273 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d1a214_2536_4b79_ac51_dc904ee807d7.slice/crio-c6e09d5e194f60bb1259dfd9596a20df0af8d40b778dc156e3f1041986c2c1f4 WatchSource:0}: Error finding container c6e09d5e194f60bb1259dfd9596a20df0af8d40b778dc156e3f1041986c2c1f4: Status 404 returned error can't find the container with id c6e09d5e194f60bb1259dfd9596a20df0af8d40b778dc156e3f1041986c2c1f4 Jul 28 00:58:27.131309 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.131279 2580 generic.go:356] "Generic (PLEG): container finished" podID="d418096c-489a-4949-b392-81238911bd1d" containerID="ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641" exitCode=0 Jul 28 00:58:27.131749 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.131329 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" Jul 28 00:58:27.131749 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.131382 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" event={"ID":"d418096c-489a-4949-b392-81238911bd1d","Type":"ContainerDied","Data":"ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641"} Jul 28 00:58:27.131749 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.131448 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84" event={"ID":"d418096c-489a-4949-b392-81238911bd1d","Type":"ContainerDied","Data":"14c3ed385e0eeddecee3fc2ec92cee52e4dfcd40eb16163cd1f7f3e680d30868"} Jul 28 00:58:27.131749 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.131494 2580 scope.go:117] "RemoveContainer" containerID="ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641" Jul 28 00:58:27.132835 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.132809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" event={"ID":"72d1a214-2536-4b79-ac51-dc904ee807d7","Type":"ContainerStarted","Data":"c6e09d5e194f60bb1259dfd9596a20df0af8d40b778dc156e3f1041986c2c1f4"} Jul 28 00:58:27.139994 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.139975 2580 scope.go:117] "RemoveContainer" containerID="ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641" Jul 28 00:58:27.140311 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:58:27.140270 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641\": container with ID starting with ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641 not found: ID does not exist" containerID="ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641" Jul 28 00:58:27.140415 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.140309 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641"} err="failed to get container status \"ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641\": rpc error: code = NotFound desc = could not find container \"ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641\": container with ID starting with ff12d8246b984e57ee230d3264fc9524168a01708df9e55d237aa6f03dd1f641 not found: ID does not exist" Jul 28 00:58:27.163822 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.163798 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84"] Jul 28 00:58:27.167528 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.167505 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vkh84"] Jul 28 00:58:27.792117 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:27.792078 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d418096c-489a-4949-b392-81238911bd1d" path="/var/lib/kubelet/pods/d418096c-489a-4949-b392-81238911bd1d/volumes" Jul 28 00:58:28.087169 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:28.087094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:58:30.148079 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:30.148042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" event={"ID":"72d1a214-2536-4b79-ac51-dc904ee807d7","Type":"ContainerStarted","Data":"39c4def3b9d0e9204fb7e65534ccfeb6c04dfb2bbd396458782f77bda6385def"} Jul 28 00:58:30.148505 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:30.148182 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:58:30.164293 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:30.164239 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" podStartSLOduration=1.648238598 podStartE2EDuration="4.164225032s" totalImagesPullingTime="2.515986434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:58:30.163284571 +0000 UTC m=+290.906792078" watchObservedRunningTime="2026-07-28 00:58:30.164225032 +0000 UTC m=+290.907732535" Jul 28 00:58:34.115587 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:34.115552 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-f2qh4" Jul 28 00:58:39.639000 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:39.638970 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 00:58:39.641271 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:39.641221 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 00:58:39.643471 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:58:39.643448 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 00:59:01.158184 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:01.158150 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 00:59:17.053594 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.053532 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:17.159431 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.159400 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:17.180347 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.180315 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7l7sl"] Jul 28 00:59:17.185063 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.185038 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" Jul 28 00:59:17.187589 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.187562 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 00:59:17.187707 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.187563 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 00:59:17.188658 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.188638 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-2pxbs\"" Jul 28 00:59:17.192705 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.192682 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7l7sl"] Jul 28 00:59:17.239677 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.239640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frpqd\" (UniqueName: \"kubernetes.io/projected/c6acc09d-a503-48ea-ba4d-a325f5ef2eb0-kube-api-access-frpqd\") pod \"kuadrant-operator-catalog-7l7sl\" (UID: \"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0\") " pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" Jul 28 00:59:17.248780 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.248753 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:17.340071 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.339990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frpqd\" (UniqueName: \"kubernetes.io/projected/c6acc09d-a503-48ea-ba4d-a325f5ef2eb0-kube-api-access-frpqd\") pod \"kuadrant-operator-catalog-7l7sl\" (UID: \"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0\") " pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" Jul 28 00:59:17.347703 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.347676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frpqd\" (UniqueName: \"kubernetes.io/projected/c6acc09d-a503-48ea-ba4d-a325f5ef2eb0-kube-api-access-frpqd\") pod \"kuadrant-operator-catalog-7l7sl\" (UID: \"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0\") " pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" Jul 28 00:59:17.496005 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.495977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" Jul 28 00:59:17.557759 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.557724 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7l7sl"] Jul 28 00:59:17.625254 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.625176 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7l7sl"] Jul 28 00:59:17.628600 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:59:17.628574 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6acc09d_a503_48ea_ba4d_a325f5ef2eb0.slice/crio-afc617a2ee4621bf5f3a1c7e309b911cac169447567fee2d31ed3233f0f29709 WatchSource:0}: Error finding container afc617a2ee4621bf5f3a1c7e309b911cac169447567fee2d31ed3233f0f29709: Status 404 returned error can't find the container with id afc617a2ee4621bf5f3a1c7e309b911cac169447567fee2d31ed3233f0f29709 Jul 28 00:59:17.630084 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.630065 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:59:17.977891 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:17.977814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:18.170518 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:18.170456 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:18.323526 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:18.323495 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" event={"ID":"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0","Type":"ContainerStarted","Data":"afc617a2ee4621bf5f3a1c7e309b911cac169447567fee2d31ed3233f0f29709"} Jul 28 00:59:20.339479 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:20.339445 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" event={"ID":"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0","Type":"ContainerStarted","Data":"e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897"} Jul 28 00:59:20.339932 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:20.339599 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" podUID="c6acc09d-a503-48ea-ba4d-a325f5ef2eb0" containerName="registry-server" containerID="cri-o://e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897" gracePeriod=2 Jul 28 00:59:20.354019 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:20.353821 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" podStartSLOduration=1.318137257 podStartE2EDuration="3.353806595s" totalImagesPullingTime="2.035669338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:59:20.353587518 +0000 UTC m=+341.097095022" watchObservedRunningTime="2026-07-28 00:59:20.353806595 +0000 UTC m=+341.097314101" Jul 28 00:59:20.574328 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:20.574305 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" Jul 28 00:59:20.661920 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:20.661849 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frpqd\" (UniqueName: \"kubernetes.io/projected/c6acc09d-a503-48ea-ba4d-a325f5ef2eb0-kube-api-access-frpqd\") pod \"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0\" (UID: \"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0\") " Jul 28 00:59:20.663937 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:20.663912 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6acc09d-a503-48ea-ba4d-a325f5ef2eb0-kube-api-access-frpqd" (OuterVolumeSpecName: "kube-api-access-frpqd") pod "c6acc09d-a503-48ea-ba4d-a325f5ef2eb0" (UID: "c6acc09d-a503-48ea-ba4d-a325f5ef2eb0"). InnerVolumeSpecName "kube-api-access-frpqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:59:20.762405 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:20.762355 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frpqd\" (UniqueName: \"kubernetes.io/projected/c6acc09d-a503-48ea-ba4d-a325f5ef2eb0-kube-api-access-frpqd\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 00:59:21.344250 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.344216 2580 generic.go:356] "Generic (PLEG): container finished" podID="c6acc09d-a503-48ea-ba4d-a325f5ef2eb0" containerID="e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897" exitCode=0 Jul 28 00:59:21.344760 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.344262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" event={"ID":"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0","Type":"ContainerDied","Data":"e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897"} Jul 28 00:59:21.344760 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.344263 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" Jul 28 00:59:21.344760 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.344302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7l7sl" event={"ID":"c6acc09d-a503-48ea-ba4d-a325f5ef2eb0","Type":"ContainerDied","Data":"afc617a2ee4621bf5f3a1c7e309b911cac169447567fee2d31ed3233f0f29709"} Jul 28 00:59:21.344760 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.344321 2580 scope.go:117] "RemoveContainer" containerID="e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897" Jul 28 00:59:21.352887 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.352869 2580 scope.go:117] "RemoveContainer" containerID="e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897" Jul 28 00:59:21.353122 ip-10-0-138-250 kubenswrapper[2580]: E0728 00:59:21.353105 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897\": container with ID starting with e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897 not found: ID does not exist" containerID="e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897" Jul 28 00:59:21.353170 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.353131 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897"} err="failed to get container status \"e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897\": rpc error: code = NotFound desc = could not find container \"e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897\": container with ID starting with e56fb60502fefde344045d9f95dd0e9f070296f25dbcd043ccf44455b202c897 not found: ID does not exist" Jul 28 00:59:21.363241 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.363220 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7l7sl"] Jul 28 00:59:21.366762 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.366744 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7l7sl"] Jul 28 00:59:21.791487 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:21.791458 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6acc09d-a503-48ea-ba4d-a325f5ef2eb0" path="/var/lib/kubelet/pods/c6acc09d-a503-48ea-ba4d-a325f5ef2eb0/volumes" Jul 28 00:59:26.133066 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:26.133034 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:26.180715 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:26.180684 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:28.088437 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:28.088403 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:28.142702 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:28.142669 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:46.382197 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:46.382163 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:49.007038 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.007006 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd"] Jul 28 00:59:49.007500 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.007270 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6acc09d-a503-48ea-ba4d-a325f5ef2eb0" containerName="registry-server" Jul 28 00:59:49.007500 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.007281 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6acc09d-a503-48ea-ba4d-a325f5ef2eb0" containerName="registry-server" Jul 28 00:59:49.007500 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.007331 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6acc09d-a503-48ea-ba4d-a325f5ef2eb0" containerName="registry-server" Jul 28 00:59:49.013384 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.013351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 00:59:49.015759 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.015737 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 00:59:49.016659 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.016638 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 00:59:49.016762 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.016643 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nx7kz\"" Jul 28 00:59:49.019692 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.019671 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd"] Jul 28 00:59:49.069457 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.069428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r845s\" (UniqueName: \"kubernetes.io/projected/f722f8cb-e5f7-4eb5-8f33-7a54523601cf-kube-api-access-r845s\") pod \"limitador-operator-controller-manager-859d7b78bd-jw2vd\" (UID: \"f722f8cb-e5f7-4eb5-8f33-7a54523601cf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 00:59:49.169791 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.169757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r845s\" (UniqueName: \"kubernetes.io/projected/f722f8cb-e5f7-4eb5-8f33-7a54523601cf-kube-api-access-r845s\") pod \"limitador-operator-controller-manager-859d7b78bd-jw2vd\" (UID: \"f722f8cb-e5f7-4eb5-8f33-7a54523601cf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 00:59:49.189750 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.189713 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r845s\" (UniqueName: \"kubernetes.io/projected/f722f8cb-e5f7-4eb5-8f33-7a54523601cf-kube-api-access-r845s\") pod \"limitador-operator-controller-manager-859d7b78bd-jw2vd\" (UID: \"f722f8cb-e5f7-4eb5-8f33-7a54523601cf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 00:59:49.324473 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.324442 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 00:59:49.451884 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:49.451853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd"] Jul 28 00:59:49.454562 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:59:49.454537 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf722f8cb_e5f7_4eb5_8f33_7a54523601cf.slice/crio-1904ee04f2b02928a93f47daca69344b451f1d7a529a8b82a1ec463e8dfd6b8f WatchSource:0}: Error finding container 1904ee04f2b02928a93f47daca69344b451f1d7a529a8b82a1ec463e8dfd6b8f: Status 404 returned error can't find the container with id 1904ee04f2b02928a93f47daca69344b451f1d7a529a8b82a1ec463e8dfd6b8f Jul 28 00:59:50.036403 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:50.036345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:50.135774 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:50.135743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:50.451608 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:50.451533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" event={"ID":"f722f8cb-e5f7-4eb5-8f33-7a54523601cf","Type":"ContainerStarted","Data":"1904ee04f2b02928a93f47daca69344b451f1d7a529a8b82a1ec463e8dfd6b8f"} Jul 28 00:59:51.457053 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:51.457023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" event={"ID":"f722f8cb-e5f7-4eb5-8f33-7a54523601cf","Type":"ContainerStarted","Data":"47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2"} Jul 28 00:59:51.457439 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:51.457122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 00:59:51.474472 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:51.474428 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" podStartSLOduration=2.037280568 podStartE2EDuration="3.474411723s" totalImagesPullingTime="1.437131155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:59:51.47293145 +0000 UTC m=+372.216438956" watchObservedRunningTime="2026-07-28 00:59:51.474411723 +0000 UTC m=+372.217919232" Jul 28 00:59:54.585566 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:54.585534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:54.990126 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:54.990045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:56.479256 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.479225 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:56.729191 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.729158 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt"] Jul 28 00:59:56.732250 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.732207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 00:59:56.734438 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.734416 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6q99n\"" Jul 28 00:59:56.739894 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.739873 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt"] Jul 28 00:59:56.829242 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.829215 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/c54edd73-f465-426b-87c6-5415206adeb7-kube-api-access-jc525\") pod \"authorino-operator-75cc8d5cb7-p8pnt\" (UID: \"c54edd73-f465-426b-87c6-5415206adeb7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 00:59:56.929875 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.929848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/c54edd73-f465-426b-87c6-5415206adeb7-kube-api-access-jc525\") pod \"authorino-operator-75cc8d5cb7-p8pnt\" (UID: \"c54edd73-f465-426b-87c6-5415206adeb7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 00:59:56.938964 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:56.938945 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc525\" (UniqueName: \"kubernetes.io/projected/c54edd73-f465-426b-87c6-5415206adeb7-kube-api-access-jc525\") pod \"authorino-operator-75cc8d5cb7-p8pnt\" (UID: \"c54edd73-f465-426b-87c6-5415206adeb7\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 00:59:57.042045 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:57.042020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 00:59:57.166890 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:57.166859 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt"] Jul 28 00:59:57.169927 ip-10-0-138-250 kubenswrapper[2580]: W0728 00:59:57.169897 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc54edd73_f465_426b_87c6_5415206adeb7.slice/crio-8ce5919464a8355af220a8084444ecb1e475fd46745c2b2067048fd166cdbc62 WatchSource:0}: Error finding container 8ce5919464a8355af220a8084444ecb1e475fd46745c2b2067048fd166cdbc62: Status 404 returned error can't find the container with id 8ce5919464a8355af220a8084444ecb1e475fd46745c2b2067048fd166cdbc62 Jul 28 00:59:57.483387 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:57.483283 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" event={"ID":"c54edd73-f465-426b-87c6-5415206adeb7","Type":"ContainerStarted","Data":"8ce5919464a8355af220a8084444ecb1e475fd46745c2b2067048fd166cdbc62"} Jul 28 00:59:58.025989 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:58.025937 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:58.816877 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:58.816848 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 00:59:59.493215 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:59.493179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" event={"ID":"c54edd73-f465-426b-87c6-5415206adeb7","Type":"ContainerStarted","Data":"305e2810a593aa26b15454498c31e254cb685d199553720eca1fcbe946345a5d"} Jul 28 00:59:59.493377 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:59.493288 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 00:59:59.508485 ip-10-0-138-250 kubenswrapper[2580]: I0728 00:59:59.508445 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" podStartSLOduration=1.323194308 podStartE2EDuration="3.508431036s" totalImagesPullingTime="2.185236728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:59:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:59:59.507177639 +0000 UTC m=+380.250685142" watchObservedRunningTime="2026-07-28 00:59:59.508431036 +0000 UTC m=+380.251938539" Jul 28 01:00:00.844714 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:00.844681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:02.305322 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:02.305283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:02.375106 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:02.375077 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:02.463387 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:02.463341 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 01:00:03.194272 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:03.194239 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:03.587664 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:03.587630 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:05.583062 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:05.583027 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:05.683538 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:05.683504 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:09.477980 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:09.477944 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:09.533404 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:09.533353 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:09.614075 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:09.614045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:09.722865 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:09.722834 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:09.935500 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:09.935463 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:10.208315 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:10.208237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:10.282686 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:10.282656 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:10.499655 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:10.499578 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 01:00:13.443083 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.443052 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd"] Jul 28 01:00:13.443545 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.443354 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" podUID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" containerName="manager" containerID="cri-o://47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2" gracePeriod=2 Jul 28 01:00:13.456338 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.456293 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd"] Jul 28 01:00:13.467181 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:00:13.467153 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f722f8cb-e5f7-4eb5-8f33-7a54523601cf\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-jw2vd\": pods \"limitador-operator-controller-manager-859d7b78bd-jw2vd\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 01:00:13.468884 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.468862 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n"] Jul 28 01:00:13.469142 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.469131 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" containerName="manager" Jul 28 01:00:13.469187 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.469144 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" containerName="manager" Jul 28 01:00:13.469221 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.469200 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" containerName="manager" Jul 28 01:00:13.471969 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.471951 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" Jul 28 01:00:13.487579 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.487555 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n"] Jul 28 01:00:13.501968 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:00:13.501943 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jw2vd\" is forbidden: User \"system:node:ip-10-0-138-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-250.ec2.internal' and this object" podUID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 01:00:13.564883 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.564851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89q58\" (UniqueName: \"kubernetes.io/projected/e23e86d8-2fc3-4db3-bd81-192d256ed4d5-kube-api-access-89q58\") pod \"limitador-operator-controller-manager-859d7b78bd-xtt6n\" (UID: \"e23e86d8-2fc3-4db3-bd81-192d256ed4d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" Jul 28 01:00:13.665696 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.665658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89q58\" (UniqueName: \"kubernetes.io/projected/e23e86d8-2fc3-4db3-bd81-192d256ed4d5-kube-api-access-89q58\") pod \"limitador-operator-controller-manager-859d7b78bd-xtt6n\" (UID: \"e23e86d8-2fc3-4db3-bd81-192d256ed4d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" Jul 28 01:00:13.673570 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.673551 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89q58\" (UniqueName: \"kubernetes.io/projected/e23e86d8-2fc3-4db3-bd81-192d256ed4d5-kube-api-access-89q58\") pod \"limitador-operator-controller-manager-859d7b78bd-xtt6n\" (UID: \"e23e86d8-2fc3-4db3-bd81-192d256ed4d5\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" Jul 28 01:00:13.687533 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.687511 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 01:00:13.689735 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:00:13.689713 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-jw2vd\" is forbidden: User \"system:node:ip-10-0-138-250.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-250.ec2.internal' and this object" podUID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 01:00:13.766185 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.766164 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r845s\" (UniqueName: \"kubernetes.io/projected/f722f8cb-e5f7-4eb5-8f33-7a54523601cf-kube-api-access-r845s\") pod \"f722f8cb-e5f7-4eb5-8f33-7a54523601cf\" (UID: \"f722f8cb-e5f7-4eb5-8f33-7a54523601cf\") " Jul 28 01:00:13.768048 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.768021 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f722f8cb-e5f7-4eb5-8f33-7a54523601cf-kube-api-access-r845s" (OuterVolumeSpecName: "kube-api-access-r845s") pod "f722f8cb-e5f7-4eb5-8f33-7a54523601cf" (UID: "f722f8cb-e5f7-4eb5-8f33-7a54523601cf"). InnerVolumeSpecName "kube-api-access-r845s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:00:13.791778 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.791757 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" path="/var/lib/kubelet/pods/f722f8cb-e5f7-4eb5-8f33-7a54523601cf/volumes" Jul 28 01:00:13.832383 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.832332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" Jul 28 01:00:13.867430 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.867400 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r845s\" (UniqueName: \"kubernetes.io/projected/f722f8cb-e5f7-4eb5-8f33-7a54523601cf-kube-api-access-r845s\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:00:13.976193 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:13.976164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n"] Jul 28 01:00:13.978909 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:00:13.978882 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23e86d8_2fc3_4db3_bd81_192d256ed4d5.slice/crio-00ec28cfa1ce14e69bd7b505253ee610404881ba865365c6c8029e0baca5d25d WatchSource:0}: Error finding container 00ec28cfa1ce14e69bd7b505253ee610404881ba865365c6c8029e0baca5d25d: Status 404 returned error can't find the container with id 00ec28cfa1ce14e69bd7b505253ee610404881ba865365c6c8029e0baca5d25d Jul 28 01:00:14.280724 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.280625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:14.382218 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.382187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:14.554574 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.554481 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" event={"ID":"e23e86d8-2fc3-4db3-bd81-192d256ed4d5","Type":"ContainerStarted","Data":"47e3993fef2670ad3c1b54d7db89382a398ec5caf4c2535fa88745c0d33e38ed"} Jul 28 01:00:14.554574 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.554538 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" event={"ID":"e23e86d8-2fc3-4db3-bd81-192d256ed4d5","Type":"ContainerStarted","Data":"00ec28cfa1ce14e69bd7b505253ee610404881ba865365c6c8029e0baca5d25d"} Jul 28 01:00:14.554574 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.554572 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" Jul 28 01:00:14.555761 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.555737 2580 generic.go:356] "Generic (PLEG): container finished" podID="f722f8cb-e5f7-4eb5-8f33-7a54523601cf" containerID="47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2" exitCode=0 Jul 28 01:00:14.555883 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.555789 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-jw2vd" Jul 28 01:00:14.555883 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.555810 2580 scope.go:117] "RemoveContainer" containerID="47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2" Jul 28 01:00:14.563710 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.563693 2580 scope.go:117] "RemoveContainer" containerID="47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2" Jul 28 01:00:14.563952 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:00:14.563934 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2\": container with ID starting with 47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2 not found: ID does not exist" containerID="47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2" Jul 28 01:00:14.563995 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.563959 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2"} err="failed to get container status \"47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2\": rpc error: code = NotFound desc = could not find container \"47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2\": container with ID starting with 47463d4846536d79a4f56726588032cfded6d34825e980da01f2295abfa11ef2 not found: ID does not exist" Jul 28 01:00:14.573816 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:14.573775 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" podStartSLOduration=1.573762195 podStartE2EDuration="1.573762195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:00:14.572719719 +0000 UTC m=+395.316227222" watchObservedRunningTime="2026-07-28 01:00:14.573762195 +0000 UTC m=+395.317269698" Jul 28 01:00:16.630239 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:16.630205 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:16.689887 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:16.689856 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:20.386274 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:20.386237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:20.486824 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:20.486791 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:20.540582 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:20.540549 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:20.993440 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:20.993358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:21.096221 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:21.096179 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:23.562503 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:23.562464 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:23.620552 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:23.620520 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:25.563020 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:25.562989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xtt6n" Jul 28 01:00:32.268868 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:32.268827 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:32.327119 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:32.327088 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:34.696546 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:34.696505 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:34.749023 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:34.748992 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:39.064552 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:39.064500 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:39.229288 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:39.229254 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:42.154753 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:42.154719 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:42.275628 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:42.275585 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:42.662458 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:42.662421 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:42.762542 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:42.762499 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:44.128970 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:44.128938 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:44.229407 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:44.229376 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:44.747560 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:44.747515 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:44.853014 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:44.852982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:51.877099 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:51.877064 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:51.975432 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:51.975399 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:54.334496 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:54.334459 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:00:54.386974 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:00:54.386939 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:00.861901 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.861851 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:01:00.866407 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.866386 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:00.868933 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.868915 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 01:01:00.869114 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.869102 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vq2f5\"" Jul 28 01:01:00.872166 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.872141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:01:00.885710 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.885692 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:01:00.926973 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.926947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9fb1fb0-0399-4b02-995c-ca2fe4a815dd-config-file\") pod \"limitador-limitador-54d6d988d5-25n22\" (UID: \"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:00.927112 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:00.927010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8kfx\" (UniqueName: \"kubernetes.io/projected/a9fb1fb0-0399-4b02-995c-ca2fe4a815dd-kube-api-access-t8kfx\") pod \"limitador-limitador-54d6d988d5-25n22\" (UID: \"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:01.027630 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:01.027604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8kfx\" (UniqueName: \"kubernetes.io/projected/a9fb1fb0-0399-4b02-995c-ca2fe4a815dd-kube-api-access-t8kfx\") pod \"limitador-limitador-54d6d988d5-25n22\" (UID: \"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:01.027756 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:01.027670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9fb1fb0-0399-4b02-995c-ca2fe4a815dd-config-file\") pod \"limitador-limitador-54d6d988d5-25n22\" (UID: \"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:01.028357 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:01.028339 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a9fb1fb0-0399-4b02-995c-ca2fe4a815dd-config-file\") pod \"limitador-limitador-54d6d988d5-25n22\" (UID: \"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:01.035773 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:01.035746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8kfx\" (UniqueName: \"kubernetes.io/projected/a9fb1fb0-0399-4b02-995c-ca2fe4a815dd-kube-api-access-t8kfx\") pod \"limitador-limitador-54d6d988d5-25n22\" (UID: \"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:01.177573 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:01.177494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:01.303967 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:01.303930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:01:01.306166 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:01:01.306142 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fb1fb0_0399_4b02_995c_ca2fe4a815dd.slice/crio-54a08808be2a366238d52e679d6e6225f3140985050fdef1725f858e11e9d517 WatchSource:0}: Error finding container 54a08808be2a366238d52e679d6e6225f3140985050fdef1725f858e11e9d517: Status 404 returned error can't find the container with id 54a08808be2a366238d52e679d6e6225f3140985050fdef1725f858e11e9d517 Jul 28 01:01:01.742844 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:01.742807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" event={"ID":"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd","Type":"ContainerStarted","Data":"54a08808be2a366238d52e679d6e6225f3140985050fdef1725f858e11e9d517"} Jul 28 01:01:04.757618 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:04.757578 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" event={"ID":"a9fb1fb0-0399-4b02-995c-ca2fe4a815dd","Type":"ContainerStarted","Data":"ab2704e1865434d5934d5cf34984976c21d51791a7e1c201c3be9ed37bc94172"} Jul 28 01:01:04.758060 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:04.757824 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:04.774645 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:04.774598 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" podStartSLOduration=2.2705209379999998 podStartE2EDuration="4.774584425s" totalImagesPullingTime="2.504063487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:01:04.772792949 +0000 UTC m=+445.516300454" watchObservedRunningTime="2026-07-28 01:01:04.774584425 +0000 UTC m=+445.518091967" Jul 28 01:01:15.763154 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:15.763117 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-25n22" Jul 28 01:01:20.468212 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:20.468178 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:20.566931 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:20.566899 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:20.932125 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:20.932090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:33.581085 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:33.581046 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:34.087768 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:34.087735 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:40.810118 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:40.810081 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:41.044048 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:41.044017 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:41.100334 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:41.100258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:41.292932 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:41.292900 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:44.750954 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:44.750926 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:44.851214 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:44.851158 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:52.280107 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:52.280073 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:01:52.381042 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:01:52.381011 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:17.505439 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:17.505402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:17.602888 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:17.602850 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:18.161883 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:18.161789 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:18.270217 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:18.270187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:18.955127 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:18.955095 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:02:19.089083 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:19.089053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:19.877904 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:19.877847 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:29.559228 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:29.559170 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:29.656956 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:29.656928 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:54.766343 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:54.766266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:02:54.813227 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:02:54.813198 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:06.355832 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:06.355794 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:06.913118 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:06.913088 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:03:13.349013 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:13.348974 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:14.211822 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:14.211792 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:03:22.350681 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:22.350647 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:23.512536 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:23.512498 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:03:24.147404 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:24.147358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:27.397820 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:27.397786 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:27.449171 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:27.449146 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:39.663510 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:39.663480 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:03:39.664043 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:39.663489 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:03:42.484193 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:42.484159 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:43.659680 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:43.659643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:44.711222 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:44.711187 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:03:53.129966 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.129932 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh"] Jul 28 01:03:53.133177 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.133156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.135863 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.135839 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 01:03:53.135980 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.135861 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 01:03:53.136526 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.136506 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pfrfw\"" Jul 28 01:03:53.136619 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.136509 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 28 01:03:53.143419 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.143400 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh"] Jul 28 01:03:53.154976 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.154952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:03:53.262382 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.262348 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.262529 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.262412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.262529 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.262469 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.262635 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.262550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.262635 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.262609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.262730 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.262641 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmt6d\" (UniqueName: \"kubernetes.io/projected/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-kube-api-access-bmt6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363454 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363454 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363686 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363686 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363686 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363841 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmt6d\" (UniqueName: \"kubernetes.io/projected/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-kube-api-access-bmt6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363895 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363865 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.363958 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363891 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.364046 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.363972 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.365830 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.365809 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.365974 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.365957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.370964 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.370934 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmt6d\" (UniqueName: \"kubernetes.io/projected/e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85-kube-api-access-bmt6d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gkpxh\" (UID: \"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.445174 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.445105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:03:53.568218 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:53.568164 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh"] Jul 28 01:03:53.570264 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:03:53.570239 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode605cb08_4588_4f5a_9a5f_1d1e7b1b2e85.slice/crio-1ae8f362b609d8f5b70ed2d9cc5ff93057fce199803b3636c06edfa218192c1d WatchSource:0}: Error finding container 1ae8f362b609d8f5b70ed2d9cc5ff93057fce199803b3636c06edfa218192c1d: Status 404 returned error can't find the container with id 1ae8f362b609d8f5b70ed2d9cc5ff93057fce199803b3636c06edfa218192c1d Jul 28 01:03:54.428755 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:54.428715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" event={"ID":"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85","Type":"ContainerStarted","Data":"1ae8f362b609d8f5b70ed2d9cc5ff93057fce199803b3636c06edfa218192c1d"} Jul 28 01:03:55.434127 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:55.434095 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" event={"ID":"e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85","Type":"ContainerStarted","Data":"44742b7cecd17603e3478a211bc2986bfa42d8f30e9e10c14151ac0c3f850a32"} Jul 28 01:03:55.450175 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:55.450128 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" podStartSLOduration=1.465605492 podStartE2EDuration="2.450113158s" totalImagesPullingTime="984.507666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:03:55.448792873 +0000 UTC m=+616.192300378" watchObservedRunningTime="2026-07-28 01:03:55.450113158 +0000 UTC m=+616.193620661" Jul 28 01:03:55.912779 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:55.912740 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:03:59.658806 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:03:59.658776 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:04:03.445677 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:03.445598 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:04:03.447855 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:03.445696 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:04:03.457849 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:03.457827 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:04:03.475695 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:03.475661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gkpxh" Jul 28 01:04:07.006735 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:07.006702 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:04:09.917475 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:09.917437 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:04:41.494678 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:41.494643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:04:57.955004 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:57.954966 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:04:58.054807 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:58.054774 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:04:59.087954 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:59.087909 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:04:59.182398 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:04:59.182360 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:05:20.278173 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:05:20.278142 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:05:27.112240 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:05:27.112195 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:05:36.713916 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:05:36.713827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:05:47.015710 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:05:47.015675 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:07:03.746865 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:03.746788 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:04.975851 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:04.975819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:24.730967 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:24.730929 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:24.783215 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:24.783187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:27.515189 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:27.515152 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:07:38.817061 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:38.817025 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:07:44.195776 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:44.195730 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:44.292781 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:44.292746 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:44.418354 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:44.418323 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:49.986286 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:49.986247 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:50.095056 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:50.095025 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:50.244330 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:50.244235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:50.355550 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:50.355506 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:55.852464 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:55.852428 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:07:55.952178 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:07:55.952150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:08:12.907768 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:08:12.907729 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:08:22.212007 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:08:22.211969 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:08:30.814162 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:08:30.814126 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:08:39.685819 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:08:39.685739 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:08:39.686544 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:08:39.686512 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:08:40.214858 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:08:40.214827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:08:50.616730 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:08:50.616695 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:09:02.312258 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:09:02.312220 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:09:13.405810 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:09:13.405774 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:09:23.408737 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:09:23.408700 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:09:48.814141 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:09:48.814106 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:09:58.011052 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:09:58.011014 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:10:14.344811 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:10:14.344706 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:10:21.217065 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:10:21.217029 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:10:32.009159 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:10:32.009118 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:10:41.711346 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:10:41.711307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:10:58.313719 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:10:58.313672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:11:07.320671 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:11:07.320632 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:11:18.612166 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:11:18.612132 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:11:28.306948 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:11:28.306910 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:11:37.315246 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:11:37.315169 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:12:32.494551 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:32.494508 2580 generic.go:356] "Generic (PLEG): container finished" podID="72d1a214-2536-4b79-ac51-dc904ee807d7" containerID="39c4def3b9d0e9204fb7e65534ccfeb6c04dfb2bbd396458782f77bda6385def" exitCode=1 Jul 28 01:12:32.495066 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:32.494589 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" event={"ID":"72d1a214-2536-4b79-ac51-dc904ee807d7","Type":"ContainerDied","Data":"39c4def3b9d0e9204fb7e65534ccfeb6c04dfb2bbd396458782f77bda6385def"} Jul 28 01:12:32.495066 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:32.495038 2580 scope.go:117] "RemoveContainer" containerID="39c4def3b9d0e9204fb7e65534ccfeb6c04dfb2bbd396458782f77bda6385def" Jul 28 01:12:32.495381 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:32.495347 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:12:33.500911 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:33.500875 2580 generic.go:356] "Generic (PLEG): container finished" podID="c54edd73-f465-426b-87c6-5415206adeb7" containerID="305e2810a593aa26b15454498c31e254cb685d199553720eca1fcbe946345a5d" exitCode=1 Jul 28 01:12:33.501344 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:33.500948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" event={"ID":"c54edd73-f465-426b-87c6-5415206adeb7","Type":"ContainerDied","Data":"305e2810a593aa26b15454498c31e254cb685d199553720eca1fcbe946345a5d"} Jul 28 01:12:33.501344 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:33.501276 2580 scope.go:117] "RemoveContainer" containerID="305e2810a593aa26b15454498c31e254cb685d199553720eca1fcbe946345a5d" Jul 28 01:12:33.502814 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:33.502792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" event={"ID":"72d1a214-2536-4b79-ac51-dc904ee807d7","Type":"ContainerStarted","Data":"e3263c109eb8597fabb4f393c617c4e93f14f745a19495b8316e5ec3cfbc45ad"} Jul 28 01:12:33.503011 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:33.502989 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 01:12:34.508014 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:34.507977 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" event={"ID":"c54edd73-f465-426b-87c6-5415206adeb7","Type":"ContainerStarted","Data":"aa587838498b32f80c2593cfd4c02f917752821c9fac73441120f447c4b9d5a9"} Jul 28 01:12:34.508419 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:34.508326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 01:12:39.155653 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:39.155606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:12:39.485985 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:39.485887 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:12:40.313257 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:40.313216 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:12:45.514573 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:45.514537 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-p8pnt" Jul 28 01:12:46.286624 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:46.286568 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:12:46.386459 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:46.386424 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:12:50.857292 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:50.857259 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:12:51.384313 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:12:51.384266 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:13:04.513118 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:13:04.513042 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ttnkm" Jul 28 01:13:29.115346 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:13:29.115309 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:13:38.213828 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:13:38.213793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:13:39.708281 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:13:39.708257 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:13:39.709473 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:13:39.709453 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:13:53.216773 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:13:53.216742 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:14:03.604863 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:03.604827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:14:16.135513 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:16.135462 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:14:16.188532 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:16.188503 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:14:16.508590 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:16.508503 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:14:24.816120 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:24.816081 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:14:35.318790 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:35.318708 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:14:45.921424 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:45.921386 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:14:55.730411 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:14:55.730361 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:15:05.613290 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:15:05.613257 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:15:18.614179 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:15:18.614138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:15:26.616229 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:15:26.616194 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:15:47.516995 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:15:47.516959 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:15:55.015464 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:15:55.015429 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:16:17.299491 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:17.299393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:16:20.323318 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:20.323282 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:16:29.131329 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.131296 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c"] Jul 28 01:16:29.134596 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.134573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.137321 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.137301 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-502777a0-kserve-self-signed-certs\"" Jul 28 01:16:29.144642 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.144618 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c"] Jul 28 01:16:29.162384 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.162345 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:16:29.196577 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.196550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tmp-dir\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.196577 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.196583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-model-cache\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.196793 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.196605 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tls-certs\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.196793 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.196665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8wt\" (UniqueName: \"kubernetes.io/projected/52e4bebe-0424-4af4-8ecf-aa4e1c938186-kube-api-access-gt8wt\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.196793 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.196708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-dshm\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.196793 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.196759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-home\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.297927 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.297892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tmp-dir\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.297927 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.297924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-model-cache\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.298156 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.297943 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tls-certs\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.298156 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.297975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8wt\" (UniqueName: \"kubernetes.io/projected/52e4bebe-0424-4af4-8ecf-aa4e1c938186-kube-api-access-gt8wt\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.298156 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.298002 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-dshm\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.298156 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.298104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-home\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.298431 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.298352 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tmp-dir\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.298431 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.298399 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-model-cache\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.298567 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.298548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-home\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.300196 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.300169 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-dshm\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.300538 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.300519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tls-certs\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.305593 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.305567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8wt\" (UniqueName: \"kubernetes.io/projected/52e4bebe-0424-4af4-8ecf-aa4e1c938186-kube-api-access-gt8wt\") pod \"e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.444006 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.443911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:29.567210 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:29.567186 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c"] Jul 28 01:16:29.569562 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:16:29.569533 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52e4bebe_0424_4af4_8ecf_aa4e1c938186.slice/crio-8a6ac665472846044d95e9699dc9d372a5dac1ece802532b51299ac36898d23b WatchSource:0}: Error finding container 8a6ac665472846044d95e9699dc9d372a5dac1ece802532b51299ac36898d23b: Status 404 returned error can't find the container with id 8a6ac665472846044d95e9699dc9d372a5dac1ece802532b51299ac36898d23b Jul 28 01:16:30.457561 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:30.457525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" event={"ID":"52e4bebe-0424-4af4-8ecf-aa4e1c938186","Type":"ContainerStarted","Data":"5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79"} Jul 28 01:16:30.457963 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:30.457569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" event={"ID":"52e4bebe-0424-4af4-8ecf-aa4e1c938186","Type":"ContainerStarted","Data":"8a6ac665472846044d95e9699dc9d372a5dac1ece802532b51299ac36898d23b"} Jul 28 01:16:30.474401 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:30.474337 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" podStartSLOduration=1.474321961 podStartE2EDuration="1.474321961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:30.47261681 +0000 UTC m=+1371.216124316" watchObservedRunningTime="2026-07-28 01:16:30.474321961 +0000 UTC m=+1371.217829464" Jul 28 01:16:34.019270 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:34.019232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:16:39.444757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:39.444692 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:39.444757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:39.444774 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:39.457443 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:39.457413 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:16:39.505017 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:16:39.504984 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:17:01.018181 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:01.018141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:17:02.021938 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:02.021905 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c"] Jul 28 01:17:02.039336 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:02.039306 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:17:04.703639 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:04.703606 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:17:17.040748 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.040684 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" podUID="52e4bebe-0424-4af4-8ecf-aa4e1c938186" containerName="main" containerID="cri-o://5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79" gracePeriod=45 Jul 28 01:17:17.214826 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.214793 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:17:17.292755 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.292699 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:17:17.403644 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.403613 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tmp-dir\") pod \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " Jul 28 01:17:17.403644 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.403651 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt8wt\" (UniqueName: \"kubernetes.io/projected/52e4bebe-0424-4af4-8ecf-aa4e1c938186-kube-api-access-gt8wt\") pod \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " Jul 28 01:17:17.403869 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.403696 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-model-cache\") pod \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " Jul 28 01:17:17.403869 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.403719 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-dshm\") pod \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " Jul 28 01:17:17.403869 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.403736 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-home\") pod \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " Jul 28 01:17:17.403869 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.403760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tls-certs\") pod \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\" (UID: \"52e4bebe-0424-4af4-8ecf-aa4e1c938186\") " Jul 28 01:17:17.404121 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.403904 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52e4bebe-0424-4af4-8ecf-aa4e1c938186" (UID: "52e4bebe-0424-4af4-8ecf-aa4e1c938186"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:17:17.404121 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.404010 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-model-cache" (OuterVolumeSpecName: "model-cache") pod "52e4bebe-0424-4af4-8ecf-aa4e1c938186" (UID: "52e4bebe-0424-4af4-8ecf-aa4e1c938186"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:17:17.404121 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.404024 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-home" (OuterVolumeSpecName: "home") pod "52e4bebe-0424-4af4-8ecf-aa4e1c938186" (UID: "52e4bebe-0424-4af4-8ecf-aa4e1c938186"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:17:17.405880 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.405860 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e4bebe-0424-4af4-8ecf-aa4e1c938186-kube-api-access-gt8wt" (OuterVolumeSpecName: "kube-api-access-gt8wt") pod "52e4bebe-0424-4af4-8ecf-aa4e1c938186" (UID: "52e4bebe-0424-4af4-8ecf-aa4e1c938186"). InnerVolumeSpecName "kube-api-access-gt8wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:17:17.405880 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.405872 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-dshm" (OuterVolumeSpecName: "dshm") pod "52e4bebe-0424-4af4-8ecf-aa4e1c938186" (UID: "52e4bebe-0424-4af4-8ecf-aa4e1c938186"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:17:17.405996 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.405966 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52e4bebe-0424-4af4-8ecf-aa4e1c938186" (UID: "52e4bebe-0424-4af4-8ecf-aa4e1c938186"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:17:17.504826 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.504784 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-model-cache\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:17:17.504826 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.504821 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-dshm\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:17:17.504826 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.504829 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-home\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:17:17.504826 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.504838 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tls-certs\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:17:17.504826 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.504846 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52e4bebe-0424-4af4-8ecf-aa4e1c938186-tmp-dir\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:17:17.505092 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.504854 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt8wt\" (UniqueName: \"kubernetes.io/projected/52e4bebe-0424-4af4-8ecf-aa4e1c938186-kube-api-access-gt8wt\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:17:17.652875 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.652788 2580 generic.go:356] "Generic (PLEG): container finished" podID="52e4bebe-0424-4af4-8ecf-aa4e1c938186" containerID="5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79" exitCode=0 Jul 28 01:17:17.652875 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.652845 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" Jul 28 01:17:17.652875 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.652868 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" event={"ID":"52e4bebe-0424-4af4-8ecf-aa4e1c938186","Type":"ContainerDied","Data":"5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79"} Jul 28 01:17:17.653117 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.652903 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c" event={"ID":"52e4bebe-0424-4af4-8ecf-aa4e1c938186","Type":"ContainerDied","Data":"8a6ac665472846044d95e9699dc9d372a5dac1ece802532b51299ac36898d23b"} Jul 28 01:17:17.653117 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.652924 2580 scope.go:117] "RemoveContainer" containerID="5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79" Jul 28 01:17:17.661658 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.661642 2580 scope.go:117] "RemoveContainer" containerID="5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79" Jul 28 01:17:17.661901 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:17:17.661884 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79\": container with ID starting with 5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79 not found: ID does not exist" containerID="5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79" Jul 28 01:17:17.661951 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.661908 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79"} err="failed to get container status \"5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79\": rpc error: code = NotFound desc = could not find container \"5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79\": container with ID starting with 5fb1239ab51b49c81104f8534dd3661ab6582c00f3d3d4b96ae96a4005a31e79 not found: ID does not exist" Jul 28 01:17:17.674021 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.674001 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c"] Jul 28 01:17:17.675709 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.675690 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-502777a0-kserve-86c498cb87-7bk4c"] Jul 28 01:17:17.791950 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:17.791913 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52e4bebe-0424-4af4-8ecf-aa4e1c938186" path="/var/lib/kubelet/pods/52e4bebe-0424-4af4-8ecf-aa4e1c938186/volumes" Jul 28 01:17:23.819974 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:23.819935 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:17:39.818916 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:39.818883 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:17:44.807288 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:44.807258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:17:53.816414 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:17:53.816383 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:18:03.017589 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:03.017553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:18:16.817011 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:16.816979 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:18:26.420294 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:26.420252 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:18:39.539294 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:39.539256 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:18:39.730907 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:39.730877 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:18:39.732918 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:39.732899 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:18:40.132843 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:40.132809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:18:48.718184 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:48.718149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:18:57.732040 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:18:57.732006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:19:10.147101 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:19:10.147027 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:19:23.291420 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:19:23.291382 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:19:33.442679 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:19:33.442639 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:19:46.755999 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:19:46.755963 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:20:11.447643 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:11.447612 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:20:20.928419 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:20.928376 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:20:22.678664 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:22.678619 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:34.753841 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:34.753758 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:20:44.450273 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:44.450241 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:47.655354 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:47.655317 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:20:48.130517 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:48.130477 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:48.265631 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:48.265599 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:48.767313 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:48.767270 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:54.470681 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:54.470643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:54.572712 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:54.572679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:54.731076 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:54.730998 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:55.065911 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:55.065874 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:55.163064 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:55.163022 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:57.495619 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:57.495584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:57.591129 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:57.591093 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:57.675329 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:57.675296 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:57.773441 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:57.773357 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:58.338152 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:58.338119 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:20:58.438330 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:20:58.438298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:21:01.322267 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:21:01.322233 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:21:57.162185 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:21:57.162145 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:21:58.398867 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:21:58.398820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:23:39.751674 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:23:39.751645 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:23:39.754714 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:23:39.754694 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:26:15.680819 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:15.680770 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:15.767174 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:15.767143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:18.427054 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:18.427013 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:26:19.592950 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:19.592910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:19.775082 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:19.775052 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:20.002989 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:20.002896 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:20.320988 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:20.320952 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:20.423042 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:20.423003 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:20.522246 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:20.522194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:20.858265 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:20.858226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:20.983685 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:20.983653 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:23.491747 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:23.491715 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:23.614023 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:23.613989 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:23.703705 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:23.703671 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:24.223382 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:24.223338 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:24.365255 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:24.365187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:31.527299 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:31.527223 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:26:33.599992 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:33.599947 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:33.701276 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:33.701244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:26:46.537859 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:26:46.537819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:28:39.772710 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:28:39.772678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:28:39.777758 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:28:39.777736 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:29:00.829802 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:00.829762 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:02.478912 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:02.478881 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:42.629133 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:42.629095 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:42.704844 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:42.704815 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:42.761997 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:42.761963 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:44.712720 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:44.712687 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:29:47.259627 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:47.259593 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:47.421899 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:47.421867 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:47.647766 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:47.647731 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:47.888488 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:47.888449 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:48.215391 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:48.215328 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:48.315860 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:48.315823 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:50.593383 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:50.593315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:50.675347 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:50.675311 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:50.747020 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:50.746991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:50.849888 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:50.849785 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:51.331336 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:51.331276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:51.427918 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:51.427877 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:29:55.723840 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:29:55.723799 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:30:06.347980 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:30:06.347934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:30:36.202023 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:30:36.201982 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:30:36.302484 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:30:36.302431 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:30:39.092866 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:30:39.092830 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:33:39.797894 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:33:39.797868 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:33:39.802992 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:33:39.802974 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:35:07.858964 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:07.858913 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:09.550586 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:09.550551 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:10.714962 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:10.714925 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:35:12.443979 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:12.443951 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:12.628181 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:12.628148 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:12.844945 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:12.844910 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:13.088217 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:13.088186 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:13.450715 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:13.450680 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:13.644264 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:13.644237 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:16.520443 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:16.520407 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:16.611285 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:16.611251 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:16.789132 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:16.789053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:16.943718 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:16.943688 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:17.422289 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:17.422256 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:17.524237 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:17.524208 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:22.524917 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:22.524879 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:35:28.827257 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:28.827219 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:44.048851 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:44.048780 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:35:44.105580 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:35:44.105554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:36:49.241837 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:36:49.241802 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:38:39.818938 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:38:39.818836 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:38:39.824603 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:38:39.824586 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:39:09.751771 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:39:09.751736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:33.606543 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:33.606508 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:34.811886 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:34.811849 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:36.722946 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:36.722907 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:40:38.193678 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:38.193649 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:38.355945 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:38.355905 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:38.607420 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:38.607386 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:39.110306 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:39.110277 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:39.199662 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:39.199631 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:39.300152 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:39.300117 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:40.710673 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:40.710640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:46.959133 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:46.959099 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:47.067800 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:47.067761 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:47.188961 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:47.188899 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:47.291889 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:47.291853 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:40:47.772383 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:40:47.772333 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:13.749403 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:13.749351 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:20.080698 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:20.080639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:20.187310 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:20.187267 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:20.287257 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:20.287215 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:36.246854 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:36.246819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:42.221899 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:42.221864 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:42.298355 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:42.298298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:42.415835 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:42.415797 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:42.503992 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:42.503883 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:53.057737 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:53.057681 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:41:54.002624 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:41:54.002588 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:00.134344 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:00.134298 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:00.237799 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:00.237751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:00.334670 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:00.334628 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:00.435186 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:00.435080 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:05.316744 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:05.316706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:42:14.217573 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:14.217534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:14.327859 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:14.327824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:14.416595 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:14.416554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:16.226921 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:16.226888 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:17.619922 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:17.619879 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:42:21.023589 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:21.023558 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:22.559665 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:22.559621 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:22.661677 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:22.661625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:23.269978 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:23.269948 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:27.914868 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:27.914831 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:42:29.473049 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:29.473016 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:29.569703 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:29.569667 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:29.731304 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:29.731220 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:30.189552 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:30.189524 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:30.252286 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:30.252246 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:33.146201 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:33.146169 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:33.289071 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:33.289045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:33.420732 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:33.420606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:33.528177 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:33.528128 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:34.080518 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:34.080472 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:34.363178 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:34.363078 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:37.580170 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:37.580138 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:38.316247 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:38.316212 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:38.418659 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:38.418627 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:38.782141 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:38.782110 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:39.419986 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:39.419951 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:42:44.494989 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:44.494956 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:44.619242 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:44.619207 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:54.918568 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:54.918533 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:55.017089 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:55.017053 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:55.080975 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:55.080940 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:42:57.720448 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:57.720412 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:42:59.909561 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:42:59.909531 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:00.330100 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:00.330063 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:04.371112 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:04.370990 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:06.314219 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:06.314183 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:06.406472 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:06.406418 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:06.657467 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:06.657358 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:06.894740 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:06.894709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:07.040277 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:07.040243 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:07.129518 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:07.129491 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:07.287855 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:07.287803 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:12.900685 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:12.900640 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:12.970682 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:12.970651 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:13.058489 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:13.058438 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:13.146829 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:13.146779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:13.275786 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:13.275758 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:18.984303 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:18.984263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:19.087771 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:19.087736 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:25.876638 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:25.876603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:26.220493 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:26.220415 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:26.431144 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:26.431112 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:29.043862 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:29.043826 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:29.139671 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:29.139636 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:29.286168 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:29.286136 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:29.858078 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:29.858018 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:29.952830 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:29.952775 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:33.506571 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:33.506535 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:35.911822 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:35.911783 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:43:39.839985 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:39.839876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:43:39.850240 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:39.850218 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:43:45.483234 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:45.483191 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:48.518706 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:48.518672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:43:53.799418 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:53.799347 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:43:55.495275 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:43:55.495244 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:44:05.627565 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:05.627530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:44:13.117872 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.117835 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8"] Jul 28 01:44:13.118341 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.118130 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e4bebe-0424-4af4-8ecf-aa4e1c938186" containerName="main" Jul 28 01:44:13.118341 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.118141 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e4bebe-0424-4af4-8ecf-aa4e1c938186" containerName="main" Jul 28 01:44:13.118341 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.118185 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e4bebe-0424-4af4-8ecf-aa4e1c938186" containerName="main" Jul 28 01:44:13.121084 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.121067 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" Jul 28 01:44:13.123593 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.123574 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:44:13.123697 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.123622 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8ph8w\"" Jul 28 01:44:13.129244 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.129221 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8"] Jul 28 01:44:13.240309 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.240274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01184c1-48cb-4724-bf5f-b40619f1740c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8\" (UID: \"a01184c1-48cb-4724-bf5f-b40619f1740c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" Jul 28 01:44:13.340965 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.340927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01184c1-48cb-4724-bf5f-b40619f1740c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8\" (UID: \"a01184c1-48cb-4724-bf5f-b40619f1740c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" Jul 28 01:44:13.341610 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.341589 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01184c1-48cb-4724-bf5f-b40619f1740c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8\" (UID: \"a01184c1-48cb-4724-bf5f-b40619f1740c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" Jul 28 01:44:13.432463 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.432359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" Jul 28 01:44:13.561744 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.561719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8"] Jul 28 01:44:13.564284 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:44:13.564261 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01184c1_48cb_4724_bf5f_b40619f1740c.slice/crio-557a97247332d148ee558b5451156df447012dfe61c2fd72ea678f923adf5428 WatchSource:0}: Error finding container 557a97247332d148ee558b5451156df447012dfe61c2fd72ea678f923adf5428: Status 404 returned error can't find the container with id 557a97247332d148ee558b5451156df447012dfe61c2fd72ea678f923adf5428 Jul 28 01:44:13.566190 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:13.566175 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:44:14.135333 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:14.135295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" event={"ID":"a01184c1-48cb-4724-bf5f-b40619f1740c","Type":"ContainerStarted","Data":"557a97247332d148ee558b5451156df447012dfe61c2fd72ea678f923adf5428"} Jul 28 01:44:16.145992 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:16.145961 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" event={"ID":"a01184c1-48cb-4724-bf5f-b40619f1740c","Type":"ContainerStarted","Data":"79a00e50fa91a80b6126efd679e2c6d3111ec60f1031e1f980db4c97b93bed1a"} Jul 28 01:44:16.161577 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:16.161533 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" podStartSLOduration=0.842711148 podStartE2EDuration="3.1615196s" totalImagesPullingTime="2.318808452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:44:16.160489698 +0000 UTC m=+3036.903997206" watchObservedRunningTime="2026-07-28 01:44:16.1615196 +0000 UTC m=+3036.905027082" Jul 28 01:44:17.074882 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.074847 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjlmd"] Jul 28 01:44:17.078012 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.077997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:17.080434 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.080415 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-2pxbs\"" Jul 28 01:44:17.086580 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.086553 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjlmd"] Jul 28 01:44:17.152165 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.152135 2580 generic.go:356] "Generic (PLEG): container finished" podID="a01184c1-48cb-4724-bf5f-b40619f1740c" containerID="79a00e50fa91a80b6126efd679e2c6d3111ec60f1031e1f980db4c97b93bed1a" exitCode=0 Jul 28 01:44:17.152544 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.152181 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" event={"ID":"a01184c1-48cb-4724-bf5f-b40619f1740c","Type":"ContainerDied","Data":"79a00e50fa91a80b6126efd679e2c6d3111ec60f1031e1f980db4c97b93bed1a"} Jul 28 01:44:17.171245 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.171210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6gt\" (UniqueName: \"kubernetes.io/projected/2730f05c-f45a-44e4-a0b6-4719289dae40-kube-api-access-md6gt\") pod \"kuadrant-operator-catalog-gjlmd\" (UID: \"2730f05c-f45a-44e4-a0b6-4719289dae40\") " pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:17.272611 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.272571 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6gt\" (UniqueName: \"kubernetes.io/projected/2730f05c-f45a-44e4-a0b6-4719289dae40-kube-api-access-md6gt\") pod \"kuadrant-operator-catalog-gjlmd\" (UID: \"2730f05c-f45a-44e4-a0b6-4719289dae40\") " pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:17.280887 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.280857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6gt\" (UniqueName: \"kubernetes.io/projected/2730f05c-f45a-44e4-a0b6-4719289dae40-kube-api-access-md6gt\") pod \"kuadrant-operator-catalog-gjlmd\" (UID: \"2730f05c-f45a-44e4-a0b6-4719289dae40\") " pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:17.387785 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.387689 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:17.514055 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:17.514030 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjlmd"] Jul 28 01:44:17.516238 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:44:17.516207 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2730f05c_f45a_44e4_a0b6_4719289dae40.slice/crio-4815b081d0c460b6caf55e1f2647f7fa8dbbb98ee4f954ef688e4fdf3db731d5 WatchSource:0}: Error finding container 4815b081d0c460b6caf55e1f2647f7fa8dbbb98ee4f954ef688e4fdf3db731d5: Status 404 returned error can't find the container with id 4815b081d0c460b6caf55e1f2647f7fa8dbbb98ee4f954ef688e4fdf3db731d5 Jul 28 01:44:18.158617 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:18.158580 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" event={"ID":"2730f05c-f45a-44e4-a0b6-4719289dae40","Type":"ContainerStarted","Data":"9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd"} Jul 28 01:44:18.159037 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:18.158631 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" event={"ID":"2730f05c-f45a-44e4-a0b6-4719289dae40","Type":"ContainerStarted","Data":"4815b081d0c460b6caf55e1f2647f7fa8dbbb98ee4f954ef688e4fdf3db731d5"} Jul 28 01:44:18.175899 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:18.175780 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" podStartSLOduration=0.771473836 podStartE2EDuration="1.175759337s" totalImagesPullingTime="404.285501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:44:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:44:18.174046189 +0000 UTC m=+3038.917553694" watchObservedRunningTime="2026-07-28 01:44:18.175759337 +0000 UTC m=+3038.919266840" Jul 28 01:44:18.276526 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:18.276498 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" Jul 28 01:44:18.381732 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:18.381703 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01184c1-48cb-4724-bf5f-b40619f1740c-maas-api-service-ca\") pod \"a01184c1-48cb-4724-bf5f-b40619f1740c\" (UID: \"a01184c1-48cb-4724-bf5f-b40619f1740c\") " Jul 28 01:44:18.382034 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:18.382013 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01184c1-48cb-4724-bf5f-b40619f1740c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a01184c1-48cb-4724-bf5f-b40619f1740c" (UID: "a01184c1-48cb-4724-bf5f-b40619f1740c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:44:18.483030 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:18.482943 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01184c1-48cb-4724-bf5f-b40619f1740c-maas-api-service-ca\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:44:19.164484 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:19.164455 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" Jul 28 01:44:19.164958 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:19.164489 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8" event={"ID":"a01184c1-48cb-4724-bf5f-b40619f1740c","Type":"ContainerDied","Data":"557a97247332d148ee558b5451156df447012dfe61c2fd72ea678f923adf5428"} Jul 28 01:44:19.164958 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:19.164533 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="557a97247332d148ee558b5451156df447012dfe61c2fd72ea678f923adf5428" Jul 28 01:44:27.316633 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:27.316593 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:44:27.388339 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:27.388309 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:27.388508 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:27.388359 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:27.410651 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:27.410621 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:28.222069 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:28.222044 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:28.255740 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:28.255714 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjlmd"] Jul 28 01:44:28.634027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:28.633991 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:44:28.706963 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:28.706934 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:44:28.768895 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:28.768854 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:44:30.209859 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:30.209821 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" podUID="2730f05c-f45a-44e4-a0b6-4719289dae40" containerName="registry-server" containerID="cri-o://9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd" gracePeriod=2 Jul 28 01:44:30.452121 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:30.452091 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:30.480027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:30.479950 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md6gt\" (UniqueName: \"kubernetes.io/projected/2730f05c-f45a-44e4-a0b6-4719289dae40-kube-api-access-md6gt\") pod \"2730f05c-f45a-44e4-a0b6-4719289dae40\" (UID: \"2730f05c-f45a-44e4-a0b6-4719289dae40\") " Jul 28 01:44:30.482045 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:30.482019 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2730f05c-f45a-44e4-a0b6-4719289dae40-kube-api-access-md6gt" (OuterVolumeSpecName: "kube-api-access-md6gt") pod "2730f05c-f45a-44e4-a0b6-4719289dae40" (UID: "2730f05c-f45a-44e4-a0b6-4719289dae40"). InnerVolumeSpecName "kube-api-access-md6gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:44:30.580804 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:30.580768 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md6gt\" (UniqueName: \"kubernetes.io/projected/2730f05c-f45a-44e4-a0b6-4719289dae40-kube-api-access-md6gt\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:44:31.214703 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.214665 2580 generic.go:356] "Generic (PLEG): container finished" podID="2730f05c-f45a-44e4-a0b6-4719289dae40" containerID="9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd" exitCode=0 Jul 28 01:44:31.214703 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.214706 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" Jul 28 01:44:31.215158 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.214745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" event={"ID":"2730f05c-f45a-44e4-a0b6-4719289dae40","Type":"ContainerDied","Data":"9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd"} Jul 28 01:44:31.215158 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.214798 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gjlmd" event={"ID":"2730f05c-f45a-44e4-a0b6-4719289dae40","Type":"ContainerDied","Data":"4815b081d0c460b6caf55e1f2647f7fa8dbbb98ee4f954ef688e4fdf3db731d5"} Jul 28 01:44:31.215158 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.214819 2580 scope.go:117] "RemoveContainer" containerID="9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd" Jul 28 01:44:31.224444 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.224428 2580 scope.go:117] "RemoveContainer" containerID="9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd" Jul 28 01:44:31.224687 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:44:31.224668 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd\": container with ID starting with 9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd not found: ID does not exist" containerID="9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd" Jul 28 01:44:31.224733 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.224696 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd"} err="failed to get container status \"9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd\": rpc error: code = NotFound desc = could not find container \"9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd\": container with ID starting with 9b1fdb24bed23db3703577e2a269080398ec2d0d38cd57248b8c1686160827fd not found: ID does not exist" Jul 28 01:44:31.234707 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.234680 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjlmd"] Jul 28 01:44:31.240885 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.240863 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gjlmd"] Jul 28 01:44:31.792588 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:31.792559 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2730f05c-f45a-44e4-a0b6-4719289dae40" path="/var/lib/kubelet/pods/2730f05c-f45a-44e4-a0b6-4719289dae40/volumes" Jul 28 01:44:33.210732 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:33.210692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:44:33.211329 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:33.211305 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8"] Jul 28 01:44:33.215118 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:33.215093 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ec1b3fa4-dl8t8"] Jul 28 01:44:33.792318 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:33.792285 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01184c1-48cb-4724-bf5f-b40619f1740c" path="/var/lib/kubelet/pods/a01184c1-48cb-4724-bf5f-b40619f1740c/volumes" Jul 28 01:44:36.506514 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:36.506481 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:44:36.920849 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:36.920815 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:44:41.633809 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:44:41.633779 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:18.122480 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:18.122447 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:18.604770 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:18.604737 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:24.247429 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:24.247396 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:24.349730 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:24.349692 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:24.540847 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:24.540810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:24.781814 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:24.781780 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:25.788591 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:25.788559 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:25.883528 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:25.883497 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:25.985425 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:25.985393 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:26.738483 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:26.738452 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:26.830168 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:26.830130 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:26.929462 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:26.929432 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:28.258938 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:28.258904 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:28.319628 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:28.319598 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:45:33.900812 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:33.900776 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:34.009269 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:34.009235 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:34.570013 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:34.569984 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:40.024186 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:40.024155 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:45:40.475443 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:40.475410 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:41.991221 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:41.991164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:51.232990 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:51.232958 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:45:51.540010 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:45:51.539975 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:12.487347 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:12.487263 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:19.128016 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:19.127977 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:19.225914 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:19.225842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:19.463541 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:19.463455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:19.838804 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:19.838773 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:20.116466 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:20.116384 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:22.978767 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:22.978732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:23.072648 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:23.072614 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:23.191347 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:23.191310 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:23.585110 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:23.585075 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:23.794605 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:23.794573 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:23.895893 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:23.895809 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:27.599418 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:27.599383 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:27.779964 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:27.779926 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:28.074995 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:28.074955 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:28.430483 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:28.430406 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:28.524093 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:28.524061 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:46:31.016328 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:31.016300 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:31.108098 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:31.108066 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:31.194853 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:31.194819 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:31.413223 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:31.413143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:31.819252 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:31.819222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:31.920275 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:31.920240 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:35.486359 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:35.486330 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:41.413610 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:41.413572 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:46:47.241042 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.241005 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9"] Jul 28 01:46:47.241457 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.241350 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2730f05c-f45a-44e4-a0b6-4719289dae40" containerName="registry-server" Jul 28 01:46:47.241457 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.241379 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2730f05c-f45a-44e4-a0b6-4719289dae40" containerName="registry-server" Jul 28 01:46:47.241457 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.241394 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01184c1-48cb-4724-bf5f-b40619f1740c" containerName="revoke-keys" Jul 28 01:46:47.241457 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.241400 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01184c1-48cb-4724-bf5f-b40619f1740c" containerName="revoke-keys" Jul 28 01:46:47.241457 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.241449 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01184c1-48cb-4724-bf5f-b40619f1740c" containerName="revoke-keys" Jul 28 01:46:47.241457 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.241456 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2730f05c-f45a-44e4-a0b6-4719289dae40" containerName="registry-server" Jul 28 01:46:47.244436 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.244420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.246977 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.246955 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2cd0f2\"/\"kube-root-ca.crt\"" Jul 28 01:46:47.247771 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.247750 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2cd0f2\"/\"test-model-2cd0f2-kserve-self-signed-certs\"" Jul 28 01:46:47.248002 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.247987 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2cd0f2\"/\"default-dockercfg-k7vd2\"" Jul 28 01:46:47.248131 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.248112 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2cd0f2\"/\"openshift-service-ca.crt\"" Jul 28 01:46:47.254843 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.254816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9"] Jul 28 01:46:47.270987 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.270962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:46:47.302721 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.302689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-model-cache\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.302721 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.302723 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-home\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.302979 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.302740 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-dshm\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.302979 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.302763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrdv\" (UniqueName: \"kubernetes.io/projected/43eab5c4-4790-4f74-bd8f-588b95039d93-kube-api-access-hnrdv\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.302979 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.302872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eab5c4-4790-4f74-bd8f-588b95039d93-tls-certs\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.302979 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.302953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-tmp-dir\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404280 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-model-cache\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404280 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-home\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404558 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-dshm\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404558 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404321 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrdv\" (UniqueName: \"kubernetes.io/projected/43eab5c4-4790-4f74-bd8f-588b95039d93-kube-api-access-hnrdv\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404558 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eab5c4-4790-4f74-bd8f-588b95039d93-tls-certs\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404558 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404407 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-tmp-dir\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404800 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404665 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-model-cache\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404800 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404733 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-home\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.404879 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.404822 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-tmp-dir\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.406583 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.406562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-dshm\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.406853 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.406833 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eab5c4-4790-4f74-bd8f-588b95039d93-tls-certs\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.413169 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.413142 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrdv\" (UniqueName: \"kubernetes.io/projected/43eab5c4-4790-4f74-bd8f-588b95039d93-kube-api-access-hnrdv\") pod \"test-model-2cd0f2-kserve-75f5996f7f-k82j9\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.553113 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.553078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:47.681562 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.681530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9"] Jul 28 01:46:47.683965 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:46:47.683932 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43eab5c4_4790_4f74_bd8f_588b95039d93.slice/crio-2e31fe6536d94bee7fe29762f30d0fee8b4afc69f2e477e2f28b74678b797a3b WatchSource:0}: Error finding container 2e31fe6536d94bee7fe29762f30d0fee8b4afc69f2e477e2f28b74678b797a3b: Status 404 returned error can't find the container with id 2e31fe6536d94bee7fe29762f30d0fee8b4afc69f2e477e2f28b74678b797a3b Jul 28 01:46:47.767073 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.767040 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" event={"ID":"43eab5c4-4790-4f74-bd8f-588b95039d93","Type":"ContainerStarted","Data":"8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75"} Jul 28 01:46:47.767341 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.767086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" event={"ID":"43eab5c4-4790-4f74-bd8f-588b95039d93","Type":"ContainerStarted","Data":"2e31fe6536d94bee7fe29762f30d0fee8b4afc69f2e477e2f28b74678b797a3b"} Jul 28 01:46:47.793680 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:47.793627 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" podStartSLOduration=0.793613315 podStartE2EDuration="793.613315ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:47.791418078 +0000 UTC m=+3188.534925584" watchObservedRunningTime="2026-07-28 01:46:47.793613315 +0000 UTC m=+3188.537120819" Jul 28 01:46:50.212798 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:50.212761 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:46:57.553564 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:57.553528 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:57.554090 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:57.553585 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:57.566330 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:57.566306 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:46:57.818213 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:46:57.818129 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:47:03.253961 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:47:03.253919 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:47:15.154000 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:47:15.153965 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:47:21.722830 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:47:21.722789 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:48:00.176744 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:00.176705 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9"] Jul 28 01:48:00.896771 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:00.896727 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:01.108639 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.108605 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:01.218056 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.217981 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr"] Jul 28 01:48:01.221063 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.221047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" Jul 28 01:48:01.223699 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.223664 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8ph8w\"" Jul 28 01:48:01.223699 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.223698 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:48:01.230016 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.229995 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr"] Jul 28 01:48:01.338632 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.338587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0902e664-06a4-481e-8858-bd6d4b9d959b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr\" (UID: \"0902e664-06a4-481e-8858-bd6d4b9d959b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" Jul 28 01:48:01.439234 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.439203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0902e664-06a4-481e-8858-bd6d4b9d959b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr\" (UID: \"0902e664-06a4-481e-8858-bd6d4b9d959b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" Jul 28 01:48:01.440064 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.440041 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0902e664-06a4-481e-8858-bd6d4b9d959b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr\" (UID: \"0902e664-06a4-481e-8858-bd6d4b9d959b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" Jul 28 01:48:01.531822 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.531792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" Jul 28 01:48:01.864478 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:01.864453 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr"] Jul 28 01:48:01.866835 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:48:01.866808 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0902e664_06a4_481e_8858_bd6d4b9d959b.slice/crio-8c11960d96b24b1f34bafdf4cbde370194060b1e00b4227fadb3e90ca4ce4454 WatchSource:0}: Error finding container 8c11960d96b24b1f34bafdf4cbde370194060b1e00b4227fadb3e90ca4ce4454: Status 404 returned error can't find the container with id 8c11960d96b24b1f34bafdf4cbde370194060b1e00b4227fadb3e90ca4ce4454 Jul 28 01:48:02.080135 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:02.080097 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" event={"ID":"0902e664-06a4-481e-8858-bd6d4b9d959b","Type":"ContainerStarted","Data":"8c11960d96b24b1f34bafdf4cbde370194060b1e00b4227fadb3e90ca4ce4454"} Jul 28 01:48:03.085955 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:03.085916 2580 generic.go:356] "Generic (PLEG): container finished" podID="0902e664-06a4-481e-8858-bd6d4b9d959b" containerID="d3728a8f449f8e11e96dd254df1adc1055a85d79034c0767d72b457b174f2ff4" exitCode=0 Jul 28 01:48:03.086332 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:03.086001 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" event={"ID":"0902e664-06a4-481e-8858-bd6d4b9d959b","Type":"ContainerDied","Data":"d3728a8f449f8e11e96dd254df1adc1055a85d79034c0767d72b457b174f2ff4"} Jul 28 01:48:04.216248 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:04.216227 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" Jul 28 01:48:04.365336 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:04.365253 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0902e664-06a4-481e-8858-bd6d4b9d959b-maas-api-service-ca\") pod \"0902e664-06a4-481e-8858-bd6d4b9d959b\" (UID: \"0902e664-06a4-481e-8858-bd6d4b9d959b\") " Jul 28 01:48:04.365635 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:04.365609 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0902e664-06a4-481e-8858-bd6d4b9d959b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0902e664-06a4-481e-8858-bd6d4b9d959b" (UID: "0902e664-06a4-481e-8858-bd6d4b9d959b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:48:04.466051 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:04.466014 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0902e664-06a4-481e-8858-bd6d4b9d959b-maas-api-service-ca\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:05.095769 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:05.095739 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" Jul 28 01:48:05.095769 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:05.095748 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr" event={"ID":"0902e664-06a4-481e-8858-bd6d4b9d959b","Type":"ContainerDied","Data":"8c11960d96b24b1f34bafdf4cbde370194060b1e00b4227fadb3e90ca4ce4454"} Jul 28 01:48:05.095976 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:05.095790 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c11960d96b24b1f34bafdf4cbde370194060b1e00b4227fadb3e90ca4ce4454" Jul 28 01:48:06.821948 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:06.821907 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:48:11.812889 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:11.812855 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:11.880776 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:11.880743 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:11.946857 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:11.946824 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:14.999719 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:14.999686 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:15.195272 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.195225 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" podUID="43eab5c4-4790-4f74-bd8f-588b95039d93" containerName="main" containerID="cri-o://8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75" gracePeriod=45 Jul 28 01:48:15.434180 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.434153 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:48:15.456769 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.456747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-home\") pod \"43eab5c4-4790-4f74-bd8f-588b95039d93\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " Jul 28 01:48:15.456911 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.456774 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnrdv\" (UniqueName: \"kubernetes.io/projected/43eab5c4-4790-4f74-bd8f-588b95039d93-kube-api-access-hnrdv\") pod \"43eab5c4-4790-4f74-bd8f-588b95039d93\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " Jul 28 01:48:15.456911 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.456815 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-model-cache\") pod \"43eab5c4-4790-4f74-bd8f-588b95039d93\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " Jul 28 01:48:15.456911 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.456857 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eab5c4-4790-4f74-bd8f-588b95039d93-tls-certs\") pod \"43eab5c4-4790-4f74-bd8f-588b95039d93\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " Jul 28 01:48:15.456911 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.456877 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-tmp-dir\") pod \"43eab5c4-4790-4f74-bd8f-588b95039d93\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " Jul 28 01:48:15.457124 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.456919 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-dshm\") pod \"43eab5c4-4790-4f74-bd8f-588b95039d93\" (UID: \"43eab5c4-4790-4f74-bd8f-588b95039d93\") " Jul 28 01:48:15.457182 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.457124 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-model-cache" (OuterVolumeSpecName: "model-cache") pod "43eab5c4-4790-4f74-bd8f-588b95039d93" (UID: "43eab5c4-4790-4f74-bd8f-588b95039d93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:48:15.457253 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.457220 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43eab5c4-4790-4f74-bd8f-588b95039d93" (UID: "43eab5c4-4790-4f74-bd8f-588b95039d93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:48:15.457353 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.457033 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-home" (OuterVolumeSpecName: "home") pod "43eab5c4-4790-4f74-bd8f-588b95039d93" (UID: "43eab5c4-4790-4f74-bd8f-588b95039d93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:48:15.459127 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.459096 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-dshm" (OuterVolumeSpecName: "dshm") pod "43eab5c4-4790-4f74-bd8f-588b95039d93" (UID: "43eab5c4-4790-4f74-bd8f-588b95039d93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:48:15.459232 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.459208 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43eab5c4-4790-4f74-bd8f-588b95039d93-kube-api-access-hnrdv" (OuterVolumeSpecName: "kube-api-access-hnrdv") pod "43eab5c4-4790-4f74-bd8f-588b95039d93" (UID: "43eab5c4-4790-4f74-bd8f-588b95039d93"). InnerVolumeSpecName "kube-api-access-hnrdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:48:15.459232 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.459206 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43eab5c4-4790-4f74-bd8f-588b95039d93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43eab5c4-4790-4f74-bd8f-588b95039d93" (UID: "43eab5c4-4790-4f74-bd8f-588b95039d93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:48:15.558014 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.557979 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-model-cache\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:15.558014 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.558010 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43eab5c4-4790-4f74-bd8f-588b95039d93-tls-certs\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:15.558014 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.558019 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-tmp-dir\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:15.558245 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.558026 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-dshm\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:15.558245 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.558034 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43eab5c4-4790-4f74-bd8f-588b95039d93-home\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:15.558245 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:15.558043 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnrdv\" (UniqueName: \"kubernetes.io/projected/43eab5c4-4790-4f74-bd8f-588b95039d93-kube-api-access-hnrdv\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:16.141492 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.141390 2580 generic.go:356] "Generic (PLEG): container finished" podID="43eab5c4-4790-4f74-bd8f-588b95039d93" containerID="8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75" exitCode=0 Jul 28 01:48:16.141492 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.141434 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" Jul 28 01:48:16.141492 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.141459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" event={"ID":"43eab5c4-4790-4f74-bd8f-588b95039d93","Type":"ContainerDied","Data":"8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75"} Jul 28 01:48:16.142028 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.141500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9" event={"ID":"43eab5c4-4790-4f74-bd8f-588b95039d93","Type":"ContainerDied","Data":"2e31fe6536d94bee7fe29762f30d0fee8b4afc69f2e477e2f28b74678b797a3b"} Jul 28 01:48:16.142028 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.141522 2580 scope.go:117] "RemoveContainer" containerID="8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75" Jul 28 01:48:16.151809 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.151786 2580 scope.go:117] "RemoveContainer" containerID="8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75" Jul 28 01:48:16.152164 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:48:16.152136 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75\": container with ID starting with 8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75 not found: ID does not exist" containerID="8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75" Jul 28 01:48:16.152269 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.152168 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75"} err="failed to get container status \"8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75\": rpc error: code = NotFound desc = could not find container \"8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75\": container with ID starting with 8c147facdea1421c042c69d066b1154c70db613ba22af8c29d9704aeb4c19b75 not found: ID does not exist" Jul 28 01:48:16.161808 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.161782 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9"] Jul 28 01:48:16.167766 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.167744 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2cd0f2/test-model-2cd0f2-kserve-75f5996f7f-k82j9"] Jul 28 01:48:16.292064 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.292032 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:16.309864 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.309833 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr"] Jul 28 01:48:16.311924 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.311896 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2cd0f2-2txdr"] Jul 28 01:48:16.691800 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:16.691760 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:17.123720 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:17.123690 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:48:17.791825 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:17.791790 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0902e664-06a4-481e-8858-bd6d4b9d959b" path="/var/lib/kubelet/pods/0902e664-06a4-481e-8858-bd6d4b9d959b/volumes" Jul 28 01:48:17.792217 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:17.792167 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43eab5c4-4790-4f74-bd8f-588b95039d93" path="/var/lib/kubelet/pods/43eab5c4-4790-4f74-bd8f-588b95039d93/volumes" Jul 28 01:48:22.514437 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:22.514401 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:22.579736 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:22.579709 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:22.842202 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:22.842168 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:23.027698 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.027670 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:23.177680 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.177594 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv"] Jul 28 01:48:23.178498 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.178473 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0902e664-06a4-481e-8858-bd6d4b9d959b" containerName="revoke-keys" Jul 28 01:48:23.178654 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.178642 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0902e664-06a4-481e-8858-bd6d4b9d959b" containerName="revoke-keys" Jul 28 01:48:23.178749 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.178740 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43eab5c4-4790-4f74-bd8f-588b95039d93" containerName="main" Jul 28 01:48:23.178819 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.178811 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="43eab5c4-4790-4f74-bd8f-588b95039d93" containerName="main" Jul 28 01:48:23.179052 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.179039 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="43eab5c4-4790-4f74-bd8f-588b95039d93" containerName="main" Jul 28 01:48:23.179142 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.179133 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0902e664-06a4-481e-8858-bd6d4b9d959b" containerName="revoke-keys" Jul 28 01:48:23.182844 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.182814 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" Jul 28 01:48:23.185554 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.185534 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8ph8w\"" Jul 28 01:48:23.185641 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.185583 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:48:23.188840 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.188820 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv"] Jul 28 01:48:23.214426 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.214392 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca427f2-6f1a-4468-8fec-04c4b4596931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv\" (UID: \"eca427f2-6f1a-4468-8fec-04c4b4596931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" Jul 28 01:48:23.314787 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.314743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca427f2-6f1a-4468-8fec-04c4b4596931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv\" (UID: \"eca427f2-6f1a-4468-8fec-04c4b4596931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" Jul 28 01:48:23.315461 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.315440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca427f2-6f1a-4468-8fec-04c4b4596931-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv\" (UID: \"eca427f2-6f1a-4468-8fec-04c4b4596931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" Jul 28 01:48:23.493706 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.493621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" Jul 28 01:48:23.825949 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:23.825923 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv"] Jul 28 01:48:23.828360 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:48:23.828332 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca427f2_6f1a_4468_8fec_04c4b4596931.slice/crio-e6ed64e4ad414a9f4ea0396b4fd6d0122c077bd7ca3b6a8ea04e97b8cd7329c5 WatchSource:0}: Error finding container e6ed64e4ad414a9f4ea0396b4fd6d0122c077bd7ca3b6a8ea04e97b8cd7329c5: Status 404 returned error can't find the container with id e6ed64e4ad414a9f4ea0396b4fd6d0122c077bd7ca3b6a8ea04e97b8cd7329c5 Jul 28 01:48:24.177525 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:24.177491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" event={"ID":"eca427f2-6f1a-4468-8fec-04c4b4596931","Type":"ContainerStarted","Data":"84d48bc631d837dcd56d9413c071bc55323edbb9b3c917282e054ea884c8625e"} Jul 28 01:48:24.177657 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:24.177552 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" event={"ID":"eca427f2-6f1a-4468-8fec-04c4b4596931","Type":"ContainerStarted","Data":"e6ed64e4ad414a9f4ea0396b4fd6d0122c077bd7ca3b6a8ea04e97b8cd7329c5"} Jul 28 01:48:24.193532 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:24.193478 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" podStartSLOduration=1.193464881 podStartE2EDuration="1.193464881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:24.192104509 +0000 UTC m=+3284.935612015" watchObservedRunningTime="2026-07-28 01:48:24.193464881 +0000 UTC m=+3284.936972385" Jul 28 01:48:25.183615 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:25.183581 2580 generic.go:356] "Generic (PLEG): container finished" podID="eca427f2-6f1a-4468-8fec-04c4b4596931" containerID="84d48bc631d837dcd56d9413c071bc55323edbb9b3c917282e054ea884c8625e" exitCode=0 Jul 28 01:48:25.183615 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:25.183619 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" event={"ID":"eca427f2-6f1a-4468-8fec-04c4b4596931","Type":"ContainerDied","Data":"84d48bc631d837dcd56d9413c071bc55323edbb9b3c917282e054ea884c8625e"} Jul 28 01:48:26.314203 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:26.314180 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" Jul 28 01:48:26.338724 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:26.338698 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca427f2-6f1a-4468-8fec-04c4b4596931-maas-api-service-ca\") pod \"eca427f2-6f1a-4468-8fec-04c4b4596931\" (UID: \"eca427f2-6f1a-4468-8fec-04c4b4596931\") " Jul 28 01:48:26.339013 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:26.338992 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca427f2-6f1a-4468-8fec-04c4b4596931-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eca427f2-6f1a-4468-8fec-04c4b4596931" (UID: "eca427f2-6f1a-4468-8fec-04c4b4596931"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:48:26.440043 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:26.439963 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eca427f2-6f1a-4468-8fec-04c4b4596931-maas-api-service-ca\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:48:27.193280 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:27.193243 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" event={"ID":"eca427f2-6f1a-4468-8fec-04c4b4596931","Type":"ContainerDied","Data":"e6ed64e4ad414a9f4ea0396b4fd6d0122c077bd7ca3b6a8ea04e97b8cd7329c5"} Jul 28 01:48:27.193440 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:27.193294 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ed64e4ad414a9f4ea0396b4fd6d0122c077bd7ca3b6a8ea04e97b8cd7329c5" Jul 28 01:48:27.193440 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:27.193255 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv" Jul 28 01:48:33.880209 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:33.880164 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:33.948737 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:33.948703 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:34.008701 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:34.008662 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:36.715752 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:36.715715 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:48:38.348778 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:38.348743 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv"] Jul 28 01:48:38.351278 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:38.351247 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f67c19-6qhrv"] Jul 28 01:48:38.352479 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:38.352458 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:38.702536 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:38.702429 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:39.791878 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:39.791850 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eca427f2-6f1a-4468-8fec-04c4b4596931" path="/var/lib/kubelet/pods/eca427f2-6f1a-4468-8fec-04c4b4596931/volumes" Jul 28 01:48:39.867619 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:39.867511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:48:39.890111 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:39.879876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:48:44.580126 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:44.580092 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:44.654049 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:44.654024 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:44.850218 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:44.850143 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:45.065254 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:45.065226 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:45.387253 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:45.387194 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:45.755843 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:45.755759 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:45.915552 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:45.915498 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:48.911183 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:48.911150 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:49.013850 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:49.013820 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:49.145128 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:49.145094 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:49.755706 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:49.755676 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:49.854710 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:49.854679 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:49.945089 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:49.945061 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:53.381204 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:53.381171 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:53.837138 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:53.837111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:54.143849 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:54.143758 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:54.325463 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:54.325434 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:56.916440 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:56.916411 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:57.026590 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:57.026507 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:57.097190 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:57.097155 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:57.233909 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:57.233828 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:48:57.276634 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:57.276603 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:57.663669 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:57.663639 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:48:57.754347 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:48:57.754315 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:01.501142 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:01.501113 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:09.469601 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.469564 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj"] Jul 28 01:49:09.469974 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.469885 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eca427f2-6f1a-4468-8fec-04c4b4596931" containerName="revoke-keys" Jul 28 01:49:09.469974 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.469897 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="eca427f2-6f1a-4468-8fec-04c4b4596931" containerName="revoke-keys" Jul 28 01:49:09.469974 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.469953 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="eca427f2-6f1a-4468-8fec-04c4b4596931" containerName="revoke-keys" Jul 28 01:49:09.472724 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.472707 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" Jul 28 01:49:09.475002 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.474977 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8ph8w\"" Jul 28 01:49:09.475137 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.475061 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:49:09.479592 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.479568 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj"] Jul 28 01:49:09.589809 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.589767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d53891-6cb1-4f2c-ac68-478f220b6037-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-64a682-npkkj\" (UID: \"f7d53891-6cb1-4f2c-ac68-478f220b6037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" Jul 28 01:49:09.690909 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.690872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d53891-6cb1-4f2c-ac68-478f220b6037-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-64a682-npkkj\" (UID: \"f7d53891-6cb1-4f2c-ac68-478f220b6037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" Jul 28 01:49:09.691510 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.691490 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d53891-6cb1-4f2c-ac68-478f220b6037-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-64a682-npkkj\" (UID: \"f7d53891-6cb1-4f2c-ac68-478f220b6037\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" Jul 28 01:49:09.784151 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.784120 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" Jul 28 01:49:09.911794 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:09.911768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj"] Jul 28 01:49:09.914328 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:49:09.914301 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d53891_6cb1_4f2c_ac68_478f220b6037.slice/crio-691b53d4430de733ad50305e5b7a0270e56cdbbbafdf401681e104dec7bb7248 WatchSource:0}: Error finding container 691b53d4430de733ad50305e5b7a0270e56cdbbbafdf401681e104dec7bb7248: Status 404 returned error can't find the container with id 691b53d4430de733ad50305e5b7a0270e56cdbbbafdf401681e104dec7bb7248 Jul 28 01:49:10.373334 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:10.373299 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" event={"ID":"f7d53891-6cb1-4f2c-ac68-478f220b6037","Type":"ContainerStarted","Data":"04740c180a41c88fde5732cceba049f6812f1d5be957a6932faa2cff692a8a31"} Jul 28 01:49:10.373525 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:10.373359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" event={"ID":"f7d53891-6cb1-4f2c-ac68-478f220b6037","Type":"ContainerStarted","Data":"691b53d4430de733ad50305e5b7a0270e56cdbbbafdf401681e104dec7bb7248"} Jul 28 01:49:10.388474 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:10.388421 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" podStartSLOduration=1.388401542 podStartE2EDuration="1.388401542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:49:10.387059424 +0000 UTC m=+3331.130566928" watchObservedRunningTime="2026-07-28 01:49:10.388401542 +0000 UTC m=+3331.131909047" Jul 28 01:49:11.317995 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:11.317959 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:49:11.378703 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:11.378668 2580 generic.go:356] "Generic (PLEG): container finished" podID="f7d53891-6cb1-4f2c-ac68-478f220b6037" containerID="04740c180a41c88fde5732cceba049f6812f1d5be957a6932faa2cff692a8a31" exitCode=0 Jul 28 01:49:11.378867 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:11.378728 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" event={"ID":"f7d53891-6cb1-4f2c-ac68-478f220b6037","Type":"ContainerDied","Data":"04740c180a41c88fde5732cceba049f6812f1d5be957a6932faa2cff692a8a31"} Jul 28 01:49:12.527079 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:12.527054 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" Jul 28 01:49:12.616711 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:12.616678 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d53891-6cb1-4f2c-ac68-478f220b6037-maas-api-service-ca\") pod \"f7d53891-6cb1-4f2c-ac68-478f220b6037\" (UID: \"f7d53891-6cb1-4f2c-ac68-478f220b6037\") " Jul 28 01:49:12.617041 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:12.617017 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d53891-6cb1-4f2c-ac68-478f220b6037-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7d53891-6cb1-4f2c-ac68-478f220b6037" (UID: "f7d53891-6cb1-4f2c-ac68-478f220b6037"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:49:12.717864 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:12.717778 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7d53891-6cb1-4f2c-ac68-478f220b6037-maas-api-service-ca\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:49:13.388247 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:13.388212 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" Jul 28 01:49:13.388247 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:13.388220 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj" event={"ID":"f7d53891-6cb1-4f2c-ac68-478f220b6037","Type":"ContainerDied","Data":"691b53d4430de733ad50305e5b7a0270e56cdbbbafdf401681e104dec7bb7248"} Jul 28 01:49:13.388499 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:13.388276 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="691b53d4430de733ad50305e5b7a0270e56cdbbbafdf401681e104dec7bb7248" Jul 28 01:49:24.785999 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:24.785957 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:24.858729 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:24.858699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:24.926559 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:24.926529 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:28.818189 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:28.818150 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:49:29.552925 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:29.552893 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj"] Jul 28 01:49:29.556133 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:29.556100 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-64a682-npkkj"] Jul 28 01:49:29.556289 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:29.556262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:29.792191 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:29.792147 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d53891-6cb1-4f2c-ac68-478f220b6037" path="/var/lib/kubelet/pods/f7d53891-6cb1-4f2c-ac68-478f220b6037/volumes" Jul 28 01:49:29.916341 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:29.916260 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:35.615091 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:35.615058 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:35.715254 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:35.715222 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:35.890661 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:35.890584 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:36.104290 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:36.104258 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:46.637639 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:46.637606 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:51.725863 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:51.725833 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:51.798410 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:51.798379 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:51.856119 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:51.856090 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:56.300964 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:56.300935 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:56.688529 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:56.688443 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:49:56.727548 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:49:56.727515 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:50:01.424997 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:01.424962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:02.409632 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:02.409598 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:02.509872 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:02.509837 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:02.694842 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:02.694751 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:02.904572 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:02.904539 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:03.264674 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:03.264643 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:03.675005 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:03.674962 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:03.986943 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:03.986845 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:06.432302 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:06.432273 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:06.524855 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:06.524814 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:06.612260 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:06.612234 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:06.966451 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:06.966420 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:07.213498 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:07.213469 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:07.321139 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:07.321105 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:10.907714 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:10.907673 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:11.209275 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:11.209181 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:11.515003 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:11.514919 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:11.651870 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:11.651838 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:11.818480 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:11.818446 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:50:14.448827 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:14.448790 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:14.547890 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:14.547859 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:14.645892 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:14.645857 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:14.906924 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:14.906889 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:15.285983 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:15.285946 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:15.386937 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:15.386895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:18.919542 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:18.919493 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:20.618746 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:20.618710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:50:32.655826 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.655733 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8"] Jul 28 01:50:32.656323 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.656077 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d53891-6cb1-4f2c-ac68-478f220b6037" containerName="revoke-keys" Jul 28 01:50:32.656323 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.656094 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d53891-6cb1-4f2c-ac68-478f220b6037" containerName="revoke-keys" Jul 28 01:50:32.656323 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.656158 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d53891-6cb1-4f2c-ac68-478f220b6037" containerName="revoke-keys" Jul 28 01:50:32.659076 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.659056 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.662516 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.662494 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-73e2fd\"/\"auth-test-model-73e2fd-kserve-self-signed-certs\"" Jul 28 01:50:32.662632 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.662521 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-73e2fd\"/\"openshift-service-ca.crt\"" Jul 28 01:50:32.662632 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.662587 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-73e2fd\"/\"default-dockercfg-7pqzg\"" Jul 28 01:50:32.662632 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.662593 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-73e2fd\"/\"kube-root-ca.crt\"" Jul 28 01:50:32.670414 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.670391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8"] Jul 28 01:50:32.688300 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.688274 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:32.784438 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.784397 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692fdb36-8504-4362-aff2-72ab505414b3-tls-certs\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.784438 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.784436 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jk2\" (UniqueName: \"kubernetes.io/projected/692fdb36-8504-4362-aff2-72ab505414b3-kube-api-access-l8jk2\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.784651 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.784456 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-tmp-dir\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.784651 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.784585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-home\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.784651 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.784625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-dshm\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.784651 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.784644 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-model-cache\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885098 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-home\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885281 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-dshm\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885281 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-model-cache\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885281 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692fdb36-8504-4362-aff2-72ab505414b3-tls-certs\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885281 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885277 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jk2\" (UniqueName: \"kubernetes.io/projected/692fdb36-8504-4362-aff2-72ab505414b3-kube-api-access-l8jk2\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885534 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-tmp-dir\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885598 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-home\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885654 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885606 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-model-cache\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.885708 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.885698 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-tmp-dir\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.887535 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.887512 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-dshm\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.887721 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.887701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692fdb36-8504-4362-aff2-72ab505414b3-tls-certs\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.892874 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.892853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jk2\" (UniqueName: \"kubernetes.io/projected/692fdb36-8504-4362-aff2-72ab505414b3-kube-api-access-l8jk2\") pod \"auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:32.968417 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:32.968294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:33.095971 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:33.095948 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8"] Jul 28 01:50:33.098450 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:50:33.098425 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod692fdb36_8504_4362_aff2_72ab505414b3.slice/crio-118ff5ff03f52534debce7db5b0f079809615f6a7063ae115da256a534602899 WatchSource:0}: Error finding container 118ff5ff03f52534debce7db5b0f079809615f6a7063ae115da256a534602899: Status 404 returned error can't find the container with id 118ff5ff03f52534debce7db5b0f079809615f6a7063ae115da256a534602899 Jul 28 01:50:33.738299 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:33.738261 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" event={"ID":"692fdb36-8504-4362-aff2-72ab505414b3","Type":"ContainerStarted","Data":"e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360"} Jul 28 01:50:33.738299 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:33.738302 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" event={"ID":"692fdb36-8504-4362-aff2-72ab505414b3","Type":"ContainerStarted","Data":"118ff5ff03f52534debce7db5b0f079809615f6a7063ae115da256a534602899"} Jul 28 01:50:33.754305 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:33.754257 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" podStartSLOduration=1.754242676 podStartE2EDuration="1.754242676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:50:33.753255151 +0000 UTC m=+3414.496762656" watchObservedRunningTime="2026-07-28 01:50:33.754242676 +0000 UTC m=+3414.497750180" Jul 28 01:50:36.111786 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:36.111754 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:50:38.960490 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:38.960455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:39.018487 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:39.018455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:50:39.965292 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:39.965263 2580 scope.go:117] "RemoveContainer" containerID="79a00e50fa91a80b6126efd679e2c6d3111ec60f1031e1f980db4c97b93bed1a" Jul 28 01:50:42.969044 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:42.969006 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:42.969454 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:42.969057 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:42.981595 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:42.981561 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:43.791929 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:43.791899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:50:45.748697 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:50:45.748666 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:51:11.243412 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.243378 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7"] Jul 28 01:51:11.246489 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.246465 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.249704 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.249674 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"openshift-service-ca.crt\"" Jul 28 01:51:11.249704 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.249691 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"default-dockercfg-pmq9v\"" Jul 28 01:51:11.249896 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.249694 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"kube-root-ca.crt\"" Jul 28 01:51:11.249896 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.249824 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"auth-test-model-7d2c59-kserve-self-signed-certs\"" Jul 28 01:51:11.254951 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.254930 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7"] Jul 28 01:51:11.271589 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.271554 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:51:11.309034 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.309004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-model-cache\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.309143 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.309038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbrv\" (UniqueName: \"kubernetes.io/projected/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-kube-api-access-hlbrv\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.309143 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.309068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-home\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.309214 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.309142 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tls-certs\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.309214 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.309191 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tmp-dir\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.309282 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.309223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-dshm\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410387 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-model-cache\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410387 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbrv\" (UniqueName: \"kubernetes.io/projected/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-kube-api-access-hlbrv\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410631 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-home\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410631 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tls-certs\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410631 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410489 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tmp-dir\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410631 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-dshm\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410851 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410795 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-model-cache\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410851 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-home\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.410955 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.410852 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tmp-dir\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.412832 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.412801 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-dshm\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.413013 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.412992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tls-certs\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.418540 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.418519 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbrv\" (UniqueName: \"kubernetes.io/projected/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-kube-api-access-hlbrv\") pod \"auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.555625 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.555591 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:11.691758 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.691734 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7"] Jul 28 01:51:11.695773 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:51:11.695739 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2f0378b_49d0_4d09_98cd_15b2db05a7d0.slice/crio-b466f802c1535e5017984487a0c475af9cd2b641757f95e8528fc01332e0de60 WatchSource:0}: Error finding container b466f802c1535e5017984487a0c475af9cd2b641757f95e8528fc01332e0de60: Status 404 returned error can't find the container with id b466f802c1535e5017984487a0c475af9cd2b641757f95e8528fc01332e0de60 Jul 28 01:51:11.891072 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.890983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" event={"ID":"c2f0378b-49d0-4d09-98cd-15b2db05a7d0","Type":"ContainerStarted","Data":"5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa"} Jul 28 01:51:11.891072 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.891023 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" event={"ID":"c2f0378b-49d0-4d09-98cd-15b2db05a7d0","Type":"ContainerStarted","Data":"b466f802c1535e5017984487a0c475af9cd2b641757f95e8528fc01332e0de60"} Jul 28 01:51:11.907028 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:11.906981 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" podStartSLOduration=0.906965375 podStartE2EDuration="906.965375ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:51:11.905746152 +0000 UTC m=+3452.649253669" watchObservedRunningTime="2026-07-28 01:51:11.906965375 +0000 UTC m=+3452.650472879" Jul 28 01:51:14.819629 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:14.819592 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:51:21.556123 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:21.556085 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:21.556537 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:21.556146 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:21.568493 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:21.568470 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:21.948581 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:21.948502 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:51:40.921278 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:51:40.921242 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:52:02.821388 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:52:02.821284 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:52:19.424355 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:52:19.424320 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:52:38.967108 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:52:38.967071 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:52:39.114098 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:52:39.114064 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:52:57.816145 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:52:57.816102 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:53:05.148971 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:53:05.148939 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:53:05.266321 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:53:05.266284 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:53:20.126582 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:53:20.126545 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:53:38.619547 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:53:38.619510 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:53:39.895624 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:53:39.895518 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:53:39.903997 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:53:39.903221 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c28hc_0afac767-a39e-4ed7-be4e-d1641fde9296/ovn-acl-logging/0.log" Jul 28 01:53:59.866500 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:53:59.866455 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:54:01.717072 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:54:01.717036 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:54:18.622667 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:54:18.622629 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:54:40.006736 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:54:40.006706 2580 scope.go:117] "RemoveContainer" containerID="84d48bc631d837dcd56d9413c071bc55323edbb9b3c917282e054ea884c8625e" Jul 28 01:54:40.014547 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:54:40.014530 2580 scope.go:117] "RemoveContainer" containerID="d3728a8f449f8e11e96dd254df1adc1055a85d79034c0767d72b457b174f2ff4" Jul 28 01:54:45.115697 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:54:45.115657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:55:02.021261 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:02.021172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:55:22.417868 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:22.417829 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:55:27.119147 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:27.119115 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8"] Jul 28 01:55:27.136952 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:27.136895 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:55:28.151783 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:28.151747 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7"] Jul 28 01:55:28.171578 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:28.171537 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:55:37.071129 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:37.071099 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:55:39.612877 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:39.612845 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:55:40.033248 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:40.033216 2580 scope.go:117] "RemoveContainer" containerID="04740c180a41c88fde5732cceba049f6812f1d5be957a6932faa2cff692a8a31" Jul 28 01:55:40.489212 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:40.489121 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:55:40.587301 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:40.587268 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:55:42.137280 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.137223 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" podUID="692fdb36-8504-4362-aff2-72ab505414b3" containerName="main" containerID="cri-o://e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360" gracePeriod=45 Jul 28 01:55:42.389629 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.389569 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:55:42.397076 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397055 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8jk2\" (UniqueName: \"kubernetes.io/projected/692fdb36-8504-4362-aff2-72ab505414b3-kube-api-access-l8jk2\") pod \"692fdb36-8504-4362-aff2-72ab505414b3\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " Jul 28 01:55:42.397132 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397114 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-dshm\") pod \"692fdb36-8504-4362-aff2-72ab505414b3\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " Jul 28 01:55:42.397169 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397133 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692fdb36-8504-4362-aff2-72ab505414b3-tls-certs\") pod \"692fdb36-8504-4362-aff2-72ab505414b3\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " Jul 28 01:55:42.397257 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397238 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-home\") pod \"692fdb36-8504-4362-aff2-72ab505414b3\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " Jul 28 01:55:42.397331 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397316 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-tmp-dir\") pod \"692fdb36-8504-4362-aff2-72ab505414b3\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " Jul 28 01:55:42.397399 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397354 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-model-cache\") pod \"692fdb36-8504-4362-aff2-72ab505414b3\" (UID: \"692fdb36-8504-4362-aff2-72ab505414b3\") " Jul 28 01:55:42.397528 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397505 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-home" (OuterVolumeSpecName: "home") pod "692fdb36-8504-4362-aff2-72ab505414b3" (UID: "692fdb36-8504-4362-aff2-72ab505414b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:42.397638 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397564 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "692fdb36-8504-4362-aff2-72ab505414b3" (UID: "692fdb36-8504-4362-aff2-72ab505414b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:42.397638 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397587 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-home\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:42.397715 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.397680 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "692fdb36-8504-4362-aff2-72ab505414b3" (UID: "692fdb36-8504-4362-aff2-72ab505414b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:42.399359 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.399330 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-dshm" (OuterVolumeSpecName: "dshm") pod "692fdb36-8504-4362-aff2-72ab505414b3" (UID: "692fdb36-8504-4362-aff2-72ab505414b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:42.399359 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.399335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692fdb36-8504-4362-aff2-72ab505414b3-kube-api-access-l8jk2" (OuterVolumeSpecName: "kube-api-access-l8jk2") pod "692fdb36-8504-4362-aff2-72ab505414b3" (UID: "692fdb36-8504-4362-aff2-72ab505414b3"). InnerVolumeSpecName "kube-api-access-l8jk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:55:42.399498 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.399393 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692fdb36-8504-4362-aff2-72ab505414b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "692fdb36-8504-4362-aff2-72ab505414b3" (UID: "692fdb36-8504-4362-aff2-72ab505414b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:55:42.498133 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.498103 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-dshm\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:42.498133 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.498129 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692fdb36-8504-4362-aff2-72ab505414b3-tls-certs\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:42.498133 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.498138 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-tmp-dir\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:42.498393 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.498147 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692fdb36-8504-4362-aff2-72ab505414b3-model-cache\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:42.498393 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:42.498155 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8jk2\" (UniqueName: \"kubernetes.io/projected/692fdb36-8504-4362-aff2-72ab505414b3-kube-api-access-l8jk2\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:43.046044 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.046015 2580 generic.go:356] "Generic (PLEG): container finished" podID="692fdb36-8504-4362-aff2-72ab505414b3" containerID="e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360" exitCode=0 Jul 28 01:55:43.046236 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.046056 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" Jul 28 01:55:43.046236 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.046067 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" event={"ID":"692fdb36-8504-4362-aff2-72ab505414b3","Type":"ContainerDied","Data":"e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360"} Jul 28 01:55:43.046236 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.046098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8" event={"ID":"692fdb36-8504-4362-aff2-72ab505414b3","Type":"ContainerDied","Data":"118ff5ff03f52534debce7db5b0f079809615f6a7063ae115da256a534602899"} Jul 28 01:55:43.046236 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.046117 2580 scope.go:117] "RemoveContainer" containerID="e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360" Jul 28 01:55:43.054758 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.054743 2580 scope.go:117] "RemoveContainer" containerID="e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360" Jul 28 01:55:43.055007 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:55:43.054991 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360\": container with ID starting with e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360 not found: ID does not exist" containerID="e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360" Jul 28 01:55:43.055048 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.055014 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360"} err="failed to get container status \"e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360\": rpc error: code = NotFound desc = could not find container \"e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360\": container with ID starting with e73c2a3d6a8003e2613e577e17a6e6afbc31c78b964ea8854e9fd97ee64c6360 not found: ID does not exist" Jul 28 01:55:43.066624 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.066598 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8"] Jul 28 01:55:43.072180 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.072159 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-73e2fd/auth-test-model-73e2fd-kserve-5b67c975ff-f9mh8"] Jul 28 01:55:43.170638 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.170608 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" podUID="c2f0378b-49d0-4d09-98cd-15b2db05a7d0" containerName="main" containerID="cri-o://5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa" gracePeriod=45 Jul 28 01:55:43.417002 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.416979 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:55:43.505171 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505138 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tls-certs\") pod \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " Jul 28 01:55:43.505355 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505176 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlbrv\" (UniqueName: \"kubernetes.io/projected/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-kube-api-access-hlbrv\") pod \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " Jul 28 01:55:43.505355 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505203 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tmp-dir\") pod \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " Jul 28 01:55:43.505355 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505252 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-model-cache\") pod \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " Jul 28 01:55:43.505355 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505301 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-home\") pod \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " Jul 28 01:55:43.505355 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505327 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-dshm\") pod \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\" (UID: \"c2f0378b-49d0-4d09-98cd-15b2db05a7d0\") " Jul 28 01:55:43.505653 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505578 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2f0378b-49d0-4d09-98cd-15b2db05a7d0" (UID: "c2f0378b-49d0-4d09-98cd-15b2db05a7d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:43.505653 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505593 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-home" (OuterVolumeSpecName: "home") pod "c2f0378b-49d0-4d09-98cd-15b2db05a7d0" (UID: "c2f0378b-49d0-4d09-98cd-15b2db05a7d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:43.505653 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.505578 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2f0378b-49d0-4d09-98cd-15b2db05a7d0" (UID: "c2f0378b-49d0-4d09-98cd-15b2db05a7d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:43.507478 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.507449 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-dshm" (OuterVolumeSpecName: "dshm") pod "c2f0378b-49d0-4d09-98cd-15b2db05a7d0" (UID: "c2f0378b-49d0-4d09-98cd-15b2db05a7d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:55:43.507567 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.507501 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2f0378b-49d0-4d09-98cd-15b2db05a7d0" (UID: "c2f0378b-49d0-4d09-98cd-15b2db05a7d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:55:43.507567 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.507533 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-kube-api-access-hlbrv" (OuterVolumeSpecName: "kube-api-access-hlbrv") pod "c2f0378b-49d0-4d09-98cd-15b2db05a7d0" (UID: "c2f0378b-49d0-4d09-98cd-15b2db05a7d0"). InnerVolumeSpecName "kube-api-access-hlbrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:55:43.606757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.606680 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-home\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:43.606757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.606702 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-dshm\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:43.606757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.606711 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tls-certs\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:43.606757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.606720 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbrv\" (UniqueName: \"kubernetes.io/projected/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-kube-api-access-hlbrv\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:43.606757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.606728 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-tmp-dir\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:43.606757 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.606736 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2f0378b-49d0-4d09-98cd-15b2db05a7d0-model-cache\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:55:43.791946 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:43.791905 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692fdb36-8504-4362-aff2-72ab505414b3" path="/var/lib/kubelet/pods/692fdb36-8504-4362-aff2-72ab505414b3/volumes" Jul 28 01:55:44.052328 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.052298 2580 generic.go:356] "Generic (PLEG): container finished" podID="c2f0378b-49d0-4d09-98cd-15b2db05a7d0" containerID="5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa" exitCode=0 Jul 28 01:55:44.052528 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.052341 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" Jul 28 01:55:44.052528 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.052392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" event={"ID":"c2f0378b-49d0-4d09-98cd-15b2db05a7d0","Type":"ContainerDied","Data":"5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa"} Jul 28 01:55:44.052528 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.052434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7" event={"ID":"c2f0378b-49d0-4d09-98cd-15b2db05a7d0","Type":"ContainerDied","Data":"b466f802c1535e5017984487a0c475af9cd2b641757f95e8528fc01332e0de60"} Jul 28 01:55:44.052528 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.052453 2580 scope.go:117] "RemoveContainer" containerID="5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa" Jul 28 01:55:44.060395 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.060382 2580 scope.go:117] "RemoveContainer" containerID="5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa" Jul 28 01:55:44.060635 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:55:44.060614 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa\": container with ID starting with 5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa not found: ID does not exist" containerID="5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa" Jul 28 01:55:44.060687 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.060642 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa"} err="failed to get container status \"5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa\": rpc error: code = NotFound desc = could not find container \"5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa\": container with ID starting with 5416e6d823ea87394752b5dce1fbc7c3b4f3ce93b3ea5bd06d114d4bc9302aaa not found: ID does not exist" Jul 28 01:55:44.067854 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.067832 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7"] Jul 28 01:55:44.072125 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.072106 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/auth-test-model-7d2c59-kserve-67cf4b6667-zrcs7"] Jul 28 01:55:44.831405 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:44.831344 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:55:45.791918 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:45.791878 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2f0378b-49d0-4d09-98cd-15b2db05a7d0" path="/var/lib/kubelet/pods/c2f0378b-49d0-4d09-98cd-15b2db05a7d0/volumes" Jul 28 01:55:46.660804 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:55:46.660769 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:01.640559 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.640520 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr"] Jul 28 01:56:01.641027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.640840 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2f0378b-49d0-4d09-98cd-15b2db05a7d0" containerName="main" Jul 28 01:56:01.641027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.640852 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2f0378b-49d0-4d09-98cd-15b2db05a7d0" containerName="main" Jul 28 01:56:01.641027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.640870 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692fdb36-8504-4362-aff2-72ab505414b3" containerName="main" Jul 28 01:56:01.641027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.640875 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="692fdb36-8504-4362-aff2-72ab505414b3" containerName="main" Jul 28 01:56:01.641027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.640920 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="692fdb36-8504-4362-aff2-72ab505414b3" containerName="main" Jul 28 01:56:01.641027 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.640927 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2f0378b-49d0-4d09-98cd-15b2db05a7d0" containerName="main" Jul 28 01:56:01.643448 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.643425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.647040 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.646975 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"kube-root-ca.crt\"" Jul 28 01:56:01.647040 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.647028 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"openshift-service-ca.crt\"" Jul 28 01:56:01.647233 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.647072 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"sub-test-model-7d2c59-kserve-self-signed-certs\"" Jul 28 01:56:01.647233 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.647104 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-7d2c59\"/\"default-dockercfg-pmq9v\"" Jul 28 01:56:01.654796 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.654775 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr"] Jul 28 01:56:01.671518 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.671488 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:01.741603 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.741559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tls-certs\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.741780 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.741609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-home\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.741780 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.741693 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjff\" (UniqueName: \"kubernetes.io/projected/cb92a578-eac7-4d67-8660-75d75cbc1ea7-kube-api-access-frjff\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.741780 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.741731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tmp-dir\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.741780 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.741756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-dshm\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.741780 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.741776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-model-cache\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842215 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-dshm\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842416 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842219 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-model-cache\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842416 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tls-certs\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842538 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-home\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842538 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjff\" (UniqueName: \"kubernetes.io/projected/cb92a578-eac7-4d67-8660-75d75cbc1ea7-kube-api-access-frjff\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842538 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842508 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tmp-dir\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842705 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842617 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-model-cache\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.842784 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.842762 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-home\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.843052 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.843035 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tmp-dir\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.844498 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.844468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-dshm\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.844770 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.844752 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tls-certs\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.851805 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.851785 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjff\" (UniqueName: \"kubernetes.io/projected/cb92a578-eac7-4d67-8660-75d75cbc1ea7-kube-api-access-frjff\") pod \"sub-test-model-7d2c59-kserve-76876f979-zgrlr\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:01.953085 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:01.952994 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:02.081627 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:02.081539 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr"] Jul 28 01:56:02.084112 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:56:02.084081 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb92a578_eac7_4d67_8660_75d75cbc1ea7.slice/crio-d52fcc15742c6a6e4fc89e309819967576395ad4d670e23153eabd688cb4736d WatchSource:0}: Error finding container d52fcc15742c6a6e4fc89e309819967576395ad4d670e23153eabd688cb4736d: Status 404 returned error can't find the container with id d52fcc15742c6a6e4fc89e309819967576395ad4d670e23153eabd688cb4736d Jul 28 01:56:02.134428 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:02.134402 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" event={"ID":"cb92a578-eac7-4d67-8660-75d75cbc1ea7","Type":"ContainerStarted","Data":"d52fcc15742c6a6e4fc89e309819967576395ad4d670e23153eabd688cb4736d"} Jul 28 01:56:03.140351 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:03.140317 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" event={"ID":"cb92a578-eac7-4d67-8660-75d75cbc1ea7","Type":"ContainerStarted","Data":"74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6"} Jul 28 01:56:03.158208 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:03.158160 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" podStartSLOduration=2.1581455800000002 podStartE2EDuration="2.15814558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:56:03.156858955 +0000 UTC m=+3743.900366460" watchObservedRunningTime="2026-07-28 01:56:03.15814558 +0000 UTC m=+3743.901653083" Jul 28 01:56:04.009380 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:04.009337 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:56:11.953667 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:11.953628 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:11.954069 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:11.953705 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:11.965938 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:11.965912 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:12.189039 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:12.189007 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:36.829277 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:36.829193 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:56:42.109789 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:42.109756 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:42.931861 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:42.931824 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr"] Jul 28 01:56:42.952047 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:42.952015 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:43.057643 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.057604 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm"] Jul 28 01:56:43.062096 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.062079 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" Jul 28 01:56:43.065022 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.065006 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8ph8w\"" Jul 28 01:56:43.065295 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.065275 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 01:56:43.069356 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.068931 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm"] Jul 28 01:56:43.101645 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.101611 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137c654e-faf3-4c30-b0b6-4f885bb14b03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm\" (UID: \"137c654e-faf3-4c30-b0b6-4f885bb14b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" Jul 28 01:56:43.202656 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.202563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137c654e-faf3-4c30-b0b6-4f885bb14b03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm\" (UID: \"137c654e-faf3-4c30-b0b6-4f885bb14b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" Jul 28 01:56:43.203212 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.203189 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137c654e-faf3-4c30-b0b6-4f885bb14b03-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm\" (UID: \"137c654e-faf3-4c30-b0b6-4f885bb14b03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" Jul 28 01:56:43.372790 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.372760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" Jul 28 01:56:43.509501 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:43.509480 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm"] Jul 28 01:56:43.511840 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:56:43.511811 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137c654e_faf3_4c30_b0b6_4f885bb14b03.slice/crio-3daec55df5aae809a7b1e1047cb30fe8bc31e33aed1f55b678bff1c3de222532 WatchSource:0}: Error finding container 3daec55df5aae809a7b1e1047cb30fe8bc31e33aed1f55b678bff1c3de222532: Status 404 returned error can't find the container with id 3daec55df5aae809a7b1e1047cb30fe8bc31e33aed1f55b678bff1c3de222532 Jul 28 01:56:44.314413 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:44.314355 2580 generic.go:356] "Generic (PLEG): container finished" podID="137c654e-faf3-4c30-b0b6-4f885bb14b03" containerID="70b808ecddcebbb0725f37782e0f9af4bb9c30779be3baddc6b1b88d344cadef" exitCode=0 Jul 28 01:56:44.314863 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:44.314458 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" event={"ID":"137c654e-faf3-4c30-b0b6-4f885bb14b03","Type":"ContainerDied","Data":"70b808ecddcebbb0725f37782e0f9af4bb9c30779be3baddc6b1b88d344cadef"} Jul 28 01:56:44.314863 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:44.314511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" event={"ID":"137c654e-faf3-4c30-b0b6-4f885bb14b03","Type":"ContainerStarted","Data":"3daec55df5aae809a7b1e1047cb30fe8bc31e33aed1f55b678bff1c3de222532"} Jul 28 01:56:45.445006 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:45.444982 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" Jul 28 01:56:45.521476 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:45.521450 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137c654e-faf3-4c30-b0b6-4f885bb14b03-maas-api-service-ca\") pod \"137c654e-faf3-4c30-b0b6-4f885bb14b03\" (UID: \"137c654e-faf3-4c30-b0b6-4f885bb14b03\") " Jul 28 01:56:45.521783 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:45.521761 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137c654e-faf3-4c30-b0b6-4f885bb14b03-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "137c654e-faf3-4c30-b0b6-4f885bb14b03" (UID: "137c654e-faf3-4c30-b0b6-4f885bb14b03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:56:45.622835 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:45.622755 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/137c654e-faf3-4c30-b0b6-4f885bb14b03-maas-api-service-ca\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:56:46.323686 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:46.323654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" event={"ID":"137c654e-faf3-4c30-b0b6-4f885bb14b03","Type":"ContainerDied","Data":"3daec55df5aae809a7b1e1047cb30fe8bc31e33aed1f55b678bff1c3de222532"} Jul 28 01:56:46.323686 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:46.323677 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm" Jul 28 01:56:46.323912 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:46.323703 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3daec55df5aae809a7b1e1047cb30fe8bc31e33aed1f55b678bff1c3de222532" Jul 28 01:56:54.236874 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:54.236841 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:54.338542 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:54.338512 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:54.434216 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:54.434182 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:56.619535 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:56.619502 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:56:57.949834 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:57.949794 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" podUID="cb92a578-eac7-4d67-8660-75d75cbc1ea7" containerName="main" containerID="cri-o://74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6" gracePeriod=45 Jul 28 01:56:58.143324 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.143283 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:58.160228 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.160201 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm"] Jul 28 01:56:58.164539 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.164509 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-73e2fd-4lpmm"] Jul 28 01:56:58.194013 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.193989 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:58.224190 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224106 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frjff\" (UniqueName: \"kubernetes.io/projected/cb92a578-eac7-4d67-8660-75d75cbc1ea7-kube-api-access-frjff\") pod \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " Jul 28 01:56:58.224190 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224150 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tls-certs\") pod \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " Jul 28 01:56:58.224190 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224183 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-home\") pod \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " Jul 28 01:56:58.224489 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224246 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tmp-dir\") pod \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " Jul 28 01:56:58.224489 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224333 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-dshm\") pod \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " Jul 28 01:56:58.224489 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224379 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-model-cache\") pod \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\" (UID: \"cb92a578-eac7-4d67-8660-75d75cbc1ea7\") " Jul 28 01:56:58.224635 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224605 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb92a578-eac7-4d67-8660-75d75cbc1ea7" (UID: "cb92a578-eac7-4d67-8660-75d75cbc1ea7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:56:58.224635 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224617 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-home" (OuterVolumeSpecName: "home") pod "cb92a578-eac7-4d67-8660-75d75cbc1ea7" (UID: "cb92a578-eac7-4d67-8660-75d75cbc1ea7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:56:58.224799 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.224770 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb92a578-eac7-4d67-8660-75d75cbc1ea7" (UID: "cb92a578-eac7-4d67-8660-75d75cbc1ea7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:56:58.226428 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.226394 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-dshm" (OuterVolumeSpecName: "dshm") pod "cb92a578-eac7-4d67-8660-75d75cbc1ea7" (UID: "cb92a578-eac7-4d67-8660-75d75cbc1ea7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:56:58.226813 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.226791 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb92a578-eac7-4d67-8660-75d75cbc1ea7" (UID: "cb92a578-eac7-4d67-8660-75d75cbc1ea7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:56:58.226813 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.226802 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb92a578-eac7-4d67-8660-75d75cbc1ea7-kube-api-access-frjff" (OuterVolumeSpecName: "kube-api-access-frjff") pod "cb92a578-eac7-4d67-8660-75d75cbc1ea7" (UID: "cb92a578-eac7-4d67-8660-75d75cbc1ea7"). InnerVolumeSpecName "kube-api-access-frjff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:56:58.324961 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.324921 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjff\" (UniqueName: \"kubernetes.io/projected/cb92a578-eac7-4d67-8660-75d75cbc1ea7-kube-api-access-frjff\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:56:58.324961 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.324953 2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tls-certs\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:56:58.324961 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.324962 2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-home\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:56:58.325206 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.324991 2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-tmp-dir\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:56:58.325206 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.324999 2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-dshm\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:56:58.325206 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.325008 2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb92a578-eac7-4d67-8660-75d75cbc1ea7-model-cache\") on node \"ip-10-0-138-250.ec2.internal\" DevicePath \"\"" Jul 28 01:56:58.373653 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.373622 2580 generic.go:356] "Generic (PLEG): container finished" podID="cb92a578-eac7-4d67-8660-75d75cbc1ea7" containerID="74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6" exitCode=0 Jul 28 01:56:58.373823 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.373666 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" Jul 28 01:56:58.373823 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.373695 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" event={"ID":"cb92a578-eac7-4d67-8660-75d75cbc1ea7","Type":"ContainerDied","Data":"74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6"} Jul 28 01:56:58.373823 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.373732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr" event={"ID":"cb92a578-eac7-4d67-8660-75d75cbc1ea7","Type":"ContainerDied","Data":"d52fcc15742c6a6e4fc89e309819967576395ad4d670e23153eabd688cb4736d"} Jul 28 01:56:58.373823 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.373751 2580 scope.go:117] "RemoveContainer" containerID="74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6" Jul 28 01:56:58.382596 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.382576 2580 scope.go:117] "RemoveContainer" containerID="74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6" Jul 28 01:56:58.382857 ip-10-0-138-250 kubenswrapper[2580]: E0728 01:56:58.382839 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6\": container with ID starting with 74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6 not found: ID does not exist" containerID="74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6" Jul 28 01:56:58.382906 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.382865 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6"} err="failed to get container status \"74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6\": rpc error: code = NotFound desc = could not find container \"74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6\": container with ID starting with 74e6c950ec938f0e2d29fa0b23aac56805a42445efb743881c9f6dcd4151dac6 not found: ID does not exist" Jul 28 01:56:58.394563 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.394537 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr"] Jul 28 01:56:58.400923 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.400904 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-7d2c59/sub-test-model-7d2c59-kserve-76876f979-zgrlr"] Jul 28 01:56:58.528868 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:58.528835 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:56:59.791668 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:59.791627 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137c654e-faf3-4c30-b0b6-4f885bb14b03" path="/var/lib/kubelet/pods/137c654e-faf3-4c30-b0b6-4f885bb14b03/volumes" Jul 28 01:56:59.792282 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:56:59.792184 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb92a578-eac7-4d67-8660-75d75cbc1ea7" path="/var/lib/kubelet/pods/cb92a578-eac7-4d67-8660-75d75cbc1ea7/volumes" Jul 28 01:57:04.563842 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:04.563806 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:04.641249 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:04.641221 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:04.910195 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:04.910111 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:05.081174 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:05.081141 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:15.810176 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:15.810140 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:15.877258 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:15.877231 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:15.939842 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:15.939810 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:18.912988 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:18.912952 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-25n22"] Jul 28 01:57:20.259446 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:20.259412 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:20.639966 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:20.639921 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:26.742313 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:26.742276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:26.844567 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:26.844534 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:27.032869 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:27.032842 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:27.249730 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:27.249699 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:32.943904 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:32.943873 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ttnkm_72d1a214-2536-4b79-ac51-dc904ee807d7/manager/1.log" Jul 28 01:57:33.411130 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:33.411099 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f2qh4_4a317d8a-cd99-4e7a-ad09-869b3978070b/manager/0.log" Jul 28 01:57:33.807000 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:33.806968 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6549c9954f-cpb7p_4b92f522-ccd5-47d2-b969-c0615b31a545/copy-manifests/0.log" Jul 28 01:57:33.813566 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:33.813541 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6549c9954f-cpb7p_4b92f522-ccd5-47d2-b969-c0615b31a545/manager/0.log" Jul 28 01:57:35.976478 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:35.976449 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-p8pnt_c54edd73-f465-426b-87c6-5415206adeb7/manager/1.log" Jul 28 01:57:36.543837 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:36.543809 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-25n22_a9fb1fb0-0399-4b02-995c-ca2fe4a815dd/limitador/0.log" Jul 28 01:57:36.659268 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:36.659239 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xtt6n_e23e86d8-2fc3-4db3-bd81-192d256ed4d5/manager/0.log" Jul 28 01:57:37.104630 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:37.104607 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f9t7x_9780ed89-46cb-4450-808c-b0ed27f1c3c9/discovery/0.log" Jul 28 01:57:37.216719 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:37.216691 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fbf9bd945-lp78h_692be0fe-0a4b-43c8-8ade-5a57b07790a4/kube-auth-proxy/0.log" Jul 28 01:57:38.570707 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:38.570675 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gkpxh_e605cb08-4588-4f5a-9a5f-1d1e7b1b2e85/main/0.log" Jul 28 01:57:43.834832 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:43.834798 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:43.902494 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:43.902460 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" limit=1024 Jul 28 01:57:46.620210 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:46.620180 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f6w4c_321b307e-eb62-4af8-beda-c5bd51c8a9ae/global-pull-secret-syncer/0.log" Jul 28 01:57:46.729081 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:46.729051 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5jtwb_297a3b21-7219-4065-9ef1-0180fd91da16/konnectivity-agent/0.log" Jul 28 01:57:46.894007 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:46.893923 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-250.ec2.internal_bf2ad10d87d1af597abc90248027ab31/haproxy/0.log" Jul 28 01:57:52.414206 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:52.414178 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-p8pnt_c54edd73-f465-426b-87c6-5415206adeb7/manager/1.log" Jul 28 01:57:52.416720 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:52.416703 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-p8pnt_c54edd73-f465-426b-87c6-5415206adeb7/manager/0.log" Jul 28 01:57:52.808230 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:52.808200 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-25n22_a9fb1fb0-0399-4b02-995c-ca2fe4a815dd/limitador/0.log" Jul 28 01:57:52.875529 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:52.875504 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xtt6n_e23e86d8-2fc3-4db3-bd81-192d256ed4d5/manager/0.log" Jul 28 01:57:55.028600 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:55.028569 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lwgx_b5b0aad6-1625-462c-a7b7-6ad9588599f6/node-exporter/0.log" Jul 28 01:57:55.049426 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:55.049387 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lwgx_b5b0aad6-1625-462c-a7b7-6ad9588599f6/kube-rbac-proxy/0.log" Jul 28 01:57:55.069264 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:55.069243 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lwgx_b5b0aad6-1625-462c-a7b7-6ad9588599f6/init-textfile/0.log" Jul 28 01:57:56.826390 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.826338 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv"] Jul 28 01:57:56.826767 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.826648 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137c654e-faf3-4c30-b0b6-4f885bb14b03" containerName="revoke-keys" Jul 28 01:57:56.826767 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.826662 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="137c654e-faf3-4c30-b0b6-4f885bb14b03" containerName="revoke-keys" Jul 28 01:57:56.826767 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.826680 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb92a578-eac7-4d67-8660-75d75cbc1ea7" containerName="main" Jul 28 01:57:56.826767 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.826685 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb92a578-eac7-4d67-8660-75d75cbc1ea7" containerName="main" Jul 28 01:57:56.826767 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.826749 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb92a578-eac7-4d67-8660-75d75cbc1ea7" containerName="main" Jul 28 01:57:56.826767 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.826760 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="137c654e-faf3-4c30-b0b6-4f885bb14b03" containerName="revoke-keys" Jul 28 01:57:56.829678 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.829657 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.832507 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.832481 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgbjn\"/\"kube-root-ca.crt\"" Jul 28 01:57:56.832608 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.832562 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hgbjn\"/\"default-dockercfg-vscst\"" Jul 28 01:57:56.833345 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.833328 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hgbjn\"/\"openshift-service-ca.crt\"" Jul 28 01:57:56.837341 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.837319 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv"] Jul 28 01:57:56.896408 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.896377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-sys\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.896408 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.896408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-lib-modules\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.896605 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.896433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-proc\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.896605 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.896554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-podres\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.896605 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.896590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkzkv\" (UniqueName: \"kubernetes.io/projected/4541bb5b-d351-4c56-a2c4-01c0130fd046-kube-api-access-lkzkv\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997167 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-podres\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997167 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997165 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkzkv\" (UniqueName: \"kubernetes.io/projected/4541bb5b-d351-4c56-a2c4-01c0130fd046-kube-api-access-lkzkv\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997412 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-sys\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997412 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997296 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-podres\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997412 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-lib-modules\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997412 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997355 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-sys\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997412 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997403 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-proc\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997659 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997457 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-proc\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:56.997659 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:56.997411 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4541bb5b-d351-4c56-a2c4-01c0130fd046-lib-modules\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:57.005793 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:57.005768 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkzkv\" (UniqueName: \"kubernetes.io/projected/4541bb5b-d351-4c56-a2c4-01c0130fd046-kube-api-access-lkzkv\") pod \"perf-node-gather-daemonset-5gzzv\" (UID: \"4541bb5b-d351-4c56-a2c4-01c0130fd046\") " pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:57.139680 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:57.139605 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:57.272709 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:57.272527 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv"] Jul 28 01:57:57.275143 ip-10-0-138-250 kubenswrapper[2580]: W0728 01:57:57.275114 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4541bb5b_d351_4c56_a2c4_01c0130fd046.slice/crio-1729e12c0730ce2784d0fac585ba3d59cba3c05bb9d16921e7e0ed2aba624227 WatchSource:0}: Error finding container 1729e12c0730ce2784d0fac585ba3d59cba3c05bb9d16921e7e0ed2aba624227: Status 404 returned error can't find the container with id 1729e12c0730ce2784d0fac585ba3d59cba3c05bb9d16921e7e0ed2aba624227 Jul 28 01:57:57.615512 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:57.615479 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" event={"ID":"4541bb5b-d351-4c56-a2c4-01c0130fd046","Type":"ContainerStarted","Data":"996d6c99c390137aef3cc552dbe801345980d63282e60aa755790f224c05a533"} Jul 28 01:57:57.615711 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:57.615533 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" event={"ID":"4541bb5b-d351-4c56-a2c4-01c0130fd046","Type":"ContainerStarted","Data":"1729e12c0730ce2784d0fac585ba3d59cba3c05bb9d16921e7e0ed2aba624227"} Jul 28 01:57:57.615711 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:57.615571 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:57:57.634531 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:57.634484 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" podStartSLOduration=1.6344702290000002 podStartE2EDuration="1.634470229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:57:57.63254472 +0000 UTC m=+3858.376052225" watchObservedRunningTime="2026-07-28 01:57:57.634470229 +0000 UTC m=+3858.377977732" Jul 28 01:57:58.099856 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:58.099828 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-txb7z_fd775c74-88f4-4574-9c7b-eae74a9fb32a/download-server/0.log" Jul 28 01:57:59.721629 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:59.721577 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-twvxb_edfa93cf-d7aa-4ba6-a01d-c60611cdfb34/dns/0.log" Jul 28 01:57:59.744571 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:59.744536 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-twvxb_edfa93cf-d7aa-4ba6-a01d-c60611cdfb34/kube-rbac-proxy/0.log" Jul 28 01:57:59.816775 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:57:59.816748 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6zj2k_6a03b76a-5d37-4008-91da-f37004123d1b/dns-node-resolver/0.log" Jul 28 01:58:00.411543 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:00.411514 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-fd8dbc94d-4df67_06587788-5cdc-4c3a-be0b-21dc0414aee8/registry/0.log" Jul 28 01:58:00.486172 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:00.486143 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rczhf_21bcb53a-99c9-484d-bb0b-f866d79b9db6/node-ca/0.log" Jul 28 01:58:01.529455 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:01.529350 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f9t7x_9780ed89-46cb-4450-808c-b0ed27f1c3c9/discovery/0.log" Jul 28 01:58:01.559241 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:01.559210 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fbf9bd945-lp78h_692be0fe-0a4b-43c8-8ade-5a57b07790a4/kube-auth-proxy/0.log" Jul 28 01:58:02.573666 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:02.573635 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l6mrb_dbaf0501-e6b9-46f1-a830-53b1a41dd7ab/serve-healthcheck-canary/0.log" Jul 28 01:58:03.252958 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:03.252932 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nk2n7_9e5a3055-47c7-4ccc-9907-a7c96db9b9a6/kube-rbac-proxy/0.log" Jul 28 01:58:03.276555 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:03.276525 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nk2n7_9e5a3055-47c7-4ccc-9907-a7c96db9b9a6/exporter/0.log" Jul 28 01:58:03.300413 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:03.300358 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nk2n7_9e5a3055-47c7-4ccc-9907-a7c96db9b9a6/extractor/0.log" Jul 28 01:58:03.629753 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:03.629726 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hgbjn/perf-node-gather-daemonset-5gzzv" Jul 28 01:58:05.536883 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:05.536842 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ttnkm_72d1a214-2536-4b79-ac51-dc904ee807d7/manager/1.log" Jul 28 01:58:05.537285 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:05.536909 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ttnkm_72d1a214-2536-4b79-ac51-dc904ee807d7/manager/0.log" Jul 28 01:58:06.021786 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:06.021748 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f2qh4_4a317d8a-cd99-4e7a-ad09-869b3978070b/manager/0.log" Jul 28 01:58:06.272882 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:06.272854 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6549c9954f-cpb7p_4b92f522-ccd5-47d2-b969-c0615b31a545/manager/0.log" Jul 28 01:58:06.297595 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:06.297572 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6549c9954f-cpb7p_4b92f522-ccd5-47d2-b969-c0615b31a545/copy-manifests/0.log" Jul 28 01:58:07.986688 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:07.986659 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f9t7x_9780ed89-46cb-4450-808c-b0ed27f1c3c9/discovery/0.log" Jul 28 01:58:07.997628 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:07.997607 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fbf9bd945-lp78h_692be0fe-0a4b-43c8-8ade-5a57b07790a4/kube-auth-proxy/0.log" Jul 28 01:58:09.526758 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:09.526716 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-f9t7x_9780ed89-46cb-4450-808c-b0ed27f1c3c9/discovery/0.log" Jul 28 01:58:09.540031 ip-10-0-138-250 kubenswrapper[2580]: I0728 01:58:09.539996 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fbf9bd945-lp78h_692be0fe-0a4b-43c8-8ade-5a57b07790a4/kube-auth-proxy/0.log"