Jul 15 21:07:35.652486 ip-10-0-136-220 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 21:07:35.652502 ip-10-0-136-220 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 21:07:35.652512 ip-10-0-136-220 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 21:07:35.652922 ip-10-0-136-220 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 21:07:45.793638 ip-10-0-136-220 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 21:07:45.793653 ip-10-0-136-220 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 77bae8d44a4845278e3f1fcde71286e2 -- Jul 15 21:09:41.721021 ip-10-0-136-220 systemd[1]: Starting Kubernetes Kubelet... Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.165636 2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170173 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170183 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170187 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170190 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170193 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170196 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170198 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:09:42.180724 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170201 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170203 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170206 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170208 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170211 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170215 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170218 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170222 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170226 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170229 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170232 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170235 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170238 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170240 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170244 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170247 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170249 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170252 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170255 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:09:42.181484 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170257 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170260 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170262 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170265 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170267 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170270 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170272 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170275 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170277 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170280 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170283 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170286 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170289 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170292 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170295 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170297 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170300 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170302 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170305 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170307 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:09:42.182257 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170309 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170312 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170314 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170317 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170320 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170322 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170325 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170327 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170329 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170332 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170334 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170337 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170340 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170343 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170345 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170348 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170350 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170353 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170355 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170359 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:09:42.183032 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170361 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170365 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170368 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170371 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170374 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170377 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170380 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170383 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170386 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170388 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170392 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170394 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170397 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170399 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170402 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170405 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170407 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170410 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170413 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:09:42.183849 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170415 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170418 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170421 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170423 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170426 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170428 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170431 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170433 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170436 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170439 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170441 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170445 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170449 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.170452 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171150 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171158 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171162 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171166 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171169 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:09:42.184594 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171177 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171180 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171182 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171185 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171187 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171190 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171192 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171195 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171198 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171200 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171203 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171205 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171209 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171213 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171216 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171219 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171222 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171224 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171227 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:09:42.185516 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171229 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171232 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171234 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171237 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171239 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171242 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171245 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171247 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171250 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171252 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171255 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171257 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171260 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171262 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171269 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171272 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171275 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171277 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171280 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171282 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:09:42.186530 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171284 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171287 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171289 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171292 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171295 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171297 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171299 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171302 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171304 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171307 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171310 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171312 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171315 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171317 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171319 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171322 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171325 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171327 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171330 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171332 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:09:42.187429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171335 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171337 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171340 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171344 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171347 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171349 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171352 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171360 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171362 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171365 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171367 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171370 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171372 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171374 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171377 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171380 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171382 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171384 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171387 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:09:42.188253 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171389 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171393 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171395 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171397 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171400 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171403 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171405 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171408 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171410 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171413 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171416 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171418 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171421 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171423 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171426 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.171428 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172634 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172640 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172643 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172646 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:09:42.189026 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172649 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172653 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172656 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172659 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172662 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172664 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172667 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172670 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172673 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172675 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172678 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172681 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172684 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172687 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172689 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172695 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172698 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172700 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172703 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172705 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:09:42.189830 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172708 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172711 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172715 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172718 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172721 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172724 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172726 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172729 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172733 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172735 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172738 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172740 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172743 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172746 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172749 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172751 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172753 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172769 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172772 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:09:42.190670 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172775 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172777 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172780 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172782 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172785 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172787 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172790 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172792 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172796 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172799 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172801 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172803 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172806 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172808 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172811 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172813 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172815 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172818 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172820 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172823 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:09:42.191525 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172825 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172828 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172831 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172834 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172836 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172839 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172841 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172844 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172848 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172850 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172853 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172856 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172858 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172861 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172863 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172866 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172868 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172870 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172874 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172876 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:09:42.192821 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172880 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172882 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172885 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172889 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172892 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172894 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172897 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172900 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172902 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172904 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172906 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172910 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172913 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172917 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172920 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.172923 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.172991 2553 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.172997 2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173004 2553 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173009 2553 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173016 2553 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 21:09:42.193577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173021 2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173026 2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173030 2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173033 2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173036 2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173039 2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173042 2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173045 2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173049 2553 flags.go:64] FLAG: --cgroup-root="" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173052 2553 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173054 2553 flags.go:64] FLAG: --client-ca-file="" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173058 2553 flags.go:64] FLAG: --cloud-provider="external" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173061 2553 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173065 2553 flags.go:64] FLAG: --cluster-domain="" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173068 2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173070 2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173073 2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173076 2553 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173080 2553 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173083 2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173086 2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173089 2553 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173092 2553 flags.go:64] FLAG: --contention-profiling="false" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173094 2553 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 21:09:42.194351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173099 2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173102 2553 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173104 2553 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173108 2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173111 2553 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173114 2553 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173116 2553 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173119 2553 flags.go:64] FLAG: --enable-server="true" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173122 2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173127 2553 flags.go:64] FLAG: --event-burst="100" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173129 2553 flags.go:64] FLAG: --event-qps="50" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173132 2553 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173135 2553 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173138 2553 flags.go:64] FLAG: --eviction-hard="" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173145 2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173148 2553 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173151 2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173154 2553 flags.go:64] FLAG: --eviction-soft="" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173157 2553 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173162 2553 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173165 2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173167 2553 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173170 2553 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173173 2553 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173175 2553 flags.go:64] FLAG: --feature-gates="" Jul 15 21:09:42.195191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173179 2553 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173182 2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173184 2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173187 2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173190 2553 flags.go:64] FLAG: --healthz-port="10248" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173193 2553 flags.go:64] FLAG: --help="false" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173196 2553 flags.go:64] FLAG: --hostname-override="ip-10-0-136-220.ec2.internal" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173200 2553 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173203 2553 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173206 2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173209 2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173212 2553 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173215 2553 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173218 2553 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173220 2553 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173223 2553 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173226 2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173229 2553 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173232 2553 flags.go:64] FLAG: --kube-reserved="" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173235 2553 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173237 2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173240 2553 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173243 2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173246 2553 flags.go:64] FLAG: --lock-file="" Jul 15 21:09:42.195956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173248 2553 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173251 2553 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173255 2553 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173260 2553 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173263 2553 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173266 2553 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173269 2553 flags.go:64] FLAG: --logging-format="text" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173271 2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173274 2553 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173277 2553 flags.go:64] FLAG: --manifest-url="" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173279 2553 flags.go:64] FLAG: --manifest-url-header="" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173283 2553 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173286 2553 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173290 2553 flags.go:64] FLAG: --max-pods="110" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173292 2553 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173297 2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173300 2553 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173302 2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173305 2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173308 2553 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173311 2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173317 2553 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173320 2553 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173323 2553 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 21:09:42.201315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173326 2553 flags.go:64] FLAG: --pod-cidr="" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173329 2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173334 2553 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173337 2553 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173340 2553 flags.go:64] FLAG: --pods-per-core="0" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173342 2553 flags.go:64] FLAG: --port="10250" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173345 2553 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173348 2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-069c166156e438891" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173351 2553 flags.go:64] FLAG: --qos-reserved="" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173353 2553 flags.go:64] FLAG: --read-only-port="10255" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173358 2553 flags.go:64] FLAG: --register-node="true" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173361 2553 flags.go:64] FLAG: --register-with-taints="" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173364 2553 flags.go:64] FLAG: --registry-burst="10" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173367 2553 flags.go:64] FLAG: --registry-qps="5" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173369 2553 flags.go:64] FLAG: --reserved-cpus="" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173372 2553 flags.go:64] FLAG: --reserved-memory="" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173376 2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173378 2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173381 2553 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173384 2553 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173387 2553 flags.go:64] FLAG: --runonce="false" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173389 2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173392 2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173395 2553 flags.go:64] FLAG: --seccomp-default="false" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173399 2553 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 21:09:42.203163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173402 2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173404 2553 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173407 2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173410 2553 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173413 2553 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173415 2553 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173418 2553 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173421 2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173424 2553 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173427 2553 flags.go:64] FLAG: --system-cgroups="" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173430 2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173435 2553 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173437 2553 flags.go:64] FLAG: --tls-cert-file="" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173440 2553 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173443 2553 flags.go:64] FLAG: --tls-min-version="" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173446 2553 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173448 2553 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173452 2553 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173455 2553 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173458 2553 flags.go:64] FLAG: --v="2" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173461 2553 flags.go:64] FLAG: --version="false" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173465 2553 flags.go:64] FLAG: --vmodule="" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173468 2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.173471 2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 21:09:42.309554 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173588 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173593 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173597 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173600 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173603 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173605 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173609 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173614 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173617 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173619 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173622 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173624 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173627 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173629 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173632 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173635 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173637 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173639 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173642 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173645 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:09:42.310279 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173647 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173650 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173652 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173655 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173657 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173661 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173664 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173666 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173669 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173671 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173673 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173676 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173678 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173681 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173683 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173685 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173688 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173690 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173693 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173696 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:09:42.311067 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173699 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173702 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173704 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173706 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173709 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173711 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173714 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173717 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173719 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173721 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173724 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173727 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173729 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173732 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173735 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173737 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173739 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173744 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173747 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173749 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:09:42.311820 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173751 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173754 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173772 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173776 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173779 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173781 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173783 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173786 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173789 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173791 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173794 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173798 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173801 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173803 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173807 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173811 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173814 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173817 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173820 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173822 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:09:42.312569 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173825 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173827 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173830 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173833 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173838 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173841 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173844 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173846 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173849 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173853 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173855 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173857 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173860 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173862 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173865 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173868 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173870 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173873 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:09:42.313354 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.173875 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.174370 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.180517 2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.180529 2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.180547 2553 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.180553 2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180608 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180614 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180617 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180620 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180623 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180625 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180628 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180631 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180634 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:09:42.314144 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180637 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180639 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180642 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180645 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180647 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180650 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180652 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180655 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180657 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180660 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180663 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180667 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180671 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180674 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180677 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180679 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180682 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180685 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180688 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:09:42.314903 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180691 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180694 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180697 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180699 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180702 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180704 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180707 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180709 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180712 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180715 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180717 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180720 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180723 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180725 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180727 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180730 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180732 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180742 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180745 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180748 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:09:42.315657 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180750 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180753 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180770 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180775 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180777 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180780 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180782 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180785 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180788 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180790 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180792 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180795 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180798 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180800 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180809 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180811 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180814 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180816 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180819 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180822 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:09:42.316449 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180824 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180827 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180831 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180833 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180836 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180838 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180841 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180843 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180846 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180849 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180857 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180861 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180864 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180868 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180872 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180875 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180877 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180880 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180883 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:09:42.317243 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180886 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180888 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180891 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180894 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180896 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180901 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180904 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180907 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180910 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180913 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180915 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.180918 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.180923 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181064 2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181068 2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:09:42.318019 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181072 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181074 2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181077 2553 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181079 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181082 2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181084 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181087 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181089 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181091 2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181101 2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181105 2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181109 2553 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181112 2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181115 2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181117 2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181120 2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181122 2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181125 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181127 2553 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181129 2553 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:09:42.318795 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181132 2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181135 2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181139 2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181143 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181146 2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181148 2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181151 2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181154 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181156 2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181159 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181161 2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181163 2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181167 2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181169 2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181172 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181174 2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181177 2553 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181179 2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181182 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:09:42.319531 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181184 2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181187 2553 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181189 2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181197 2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181200 2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181202 2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181205 2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181207 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181210 2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181212 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181215 2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181217 2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181220 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181222 2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181225 2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181227 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181231 2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181234 2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181236 2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:09:42.320306 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181239 2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181241 2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181244 2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181246 2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181249 2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181251 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181255 2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181257 2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181260 2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181262 2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181265 2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181267 2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181269 2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181272 2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181274 2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181277 2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181279 2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181289 2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181291 2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181293 2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:09:42.321068 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181296 2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181299 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181301 2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181303 2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181305 2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181308 2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181310 2553 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181313 2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181315 2553 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181318 2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181321 2553 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181324 2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181326 2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181328 2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181331 2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181333 2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181336 2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181339 2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:09:42.321840 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:42.181344 2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.181350 2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.181445 2553 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.183429 2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.184776 2553 server.go:1019] "Starting client certificate rotation" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.184900 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.184946 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.190506 2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.194445 2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.206458 2553 log.go:25] "Validated CRI v1 runtime API" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.208999 2553 log.go:25] "Validated CRI v1 image API" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.209747 2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 21:09:42.322603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.213237 2553 fs.go:135] Filesystem UUIDs: map[4e4e00bf-3e96-46ca-b79c-f02bfdbc188f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aadce237-2674-40fa-9298-594230050265:/dev/nvme0n1p3] Jul 15 21:09:42.322994 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.213257 2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 21:09:42.323374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.220281 2553 manager.go:216] Machine: {Timestamp:2026-07-15 21:09:42.218806808 +0000 UTC m=+0.345470783 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201088 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec253adfeb09e02a6a14996ea178d3f2 SystemUUID:ec253adf-eb09-e02a-6a14-996ea178d3f2 BootID:77bae8d4-4a48-4527-8e3f-1fcde71286e2 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:dc:c6:89:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:dc:c6:89:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:53:36:cd:d9:ee Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 21:09:42.323374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.220400 2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 21:09:42.323374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.220485 2553 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 21:09:42.323374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.220991 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 21:09:42.323374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.223678 2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 21:09:42.323374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.223702 2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-220.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 21:09:42.323374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.224497 2553 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.224505 2553 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.224512 2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.225279 2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.226468 2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230182 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230299 2553 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230323 2553 kubelet.go:486] "Attempting to sync node with API server" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230332 2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230341 2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230349 2553 kubelet.go:398] "Adding apiserver pod source" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230358 2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230694 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.230709 2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.232842 2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.234869 2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.234906 2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235078 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235093 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235102 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235109 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235122 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235130 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235584 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235593 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235602 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.235610 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.236613 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.236647 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.236679 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.236689 2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.246768 2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-220.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:42.246824 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:42.246823 2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.284269 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-btn2m" Jul 15 21:09:42.328754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.299050 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-btn2m" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.905566 2553 server.go:1268] "Started kubelet" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.905654 2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.905708 2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.905731 2553 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.906887 2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.907787 2553 server.go:310] "Adding debug handlers to kubelet server" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:42.912220 2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:42.911496 2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-220.ec2.internal.18c292c6542c67dd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-220.ec2.internal,UID:ip-10-0-136-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-220.ec2.internal,},FirstTimestamp:2026-07-15 21:09:42.905513949 +0000 UTC m=+1.032177923,LastTimestamp:2026-07-15 21:09:42.905513949 +0000 UTC m=+1.032177923,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-220.ec2.internal,}" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.913276 2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.913295 2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:42.913350 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914029 2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914043 2553 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914062 2553 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914173 2553 reconstruct.go:97] "Volume reconstruction finished" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914183 2553 reconciler.go:29] "Reconciler: start to sync state" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914205 2553 factory.go:55] Registering systemd factory Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914220 2553 factory.go:223] Registration of the systemd container factory successfully Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914500 2553 factory.go:153] Registering CRI-O factory Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914513 2553 factory.go:223] Registration of the crio container factory successfully Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914560 2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914584 2553 factory.go:103] Registering Raw factory Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914598 2553 manager.go:1196] Started watching for new ooms in manager Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.914999 2553 manager.go:349] Starting recovery of all containers Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.927767 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:42.931480 2553 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-220.ec2.internal\" not found" node="ip-10-0-136-220.ec2.internal" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.933405 2553 manager.go:354] Recovery completed Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.941782 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.944356 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.944396 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.944412 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.945099 2553 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.945107 2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.945166 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.945322 2553 state_mem.go:36] "Initialized new in-memory state store" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.946658 2553 policy_none.go:49] "None policy: Start" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.946671 2553 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 21:09:43.022392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.946682 2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 21:09:42.906596 ip-10-0-136-220 systemd[1]: Started Kubernetes Kubelet. Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.947964 2553 policy_none.go:47] "Start" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.949572 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.949599 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:42.949614 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.002367 2553 manager.go:337] "Starting Device Plugin manager" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.002503 2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.002513 2553 server.go:87] "Starting device plugin registration server" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.002723 2553 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.002735 2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.003787 2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.003826 2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.008653 2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.008725 2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 21:09:43.023986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.008731 2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.103032 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.103927 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.103945 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.103955 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.103979 2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-220.ec2.internal" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.112852 2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-220.ec2.internal" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.112868 2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-220.ec2.internal\": node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.129185 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.136229 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.137311 2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.137336 2553 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.137354 2553 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.137381 2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 21:09:43.155976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.142076 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.230252 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.237499 2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal"] Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.237546 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.238278 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.238295 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.238311 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.239478 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.239779 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.239821 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.240418 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.240435 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.240444 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.241754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.241774 2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.241904 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.241919 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.241927 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.242665 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.242680 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.242688 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.247239 2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.247330 2553 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a1d87dc9d67704d52a2077ed9e14b9be-e0a1f30c8186735e.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-136-220.ec2.internal.18c292c6567d8bdc\": read tcp 10.0.136.220:44846->3.223.202.132:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-136-220.ec2.internal.18c292c6567d8bdc default 6434 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-220.ec2.internal,UID:ip-10-0-136-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-220.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-220.ec2.internal,},FirstTimestamp:2026-07-15 21:09:42 +0000 UTC,LastTimestamp:2026-07-15 21:09:43.238286541 +0000 UTC m=+1.364950515,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-220.ec2.internal,}" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.247411 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 21:09:43.292335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.247434 2553 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 21:09:43.418425 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.330862 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.418425 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:43.370734 2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:09:43.439334 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.431936 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.541713 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.532175 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.668851 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.632465 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.790938 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.732809 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.898382 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.832915 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:43.933167 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:43.933149 2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-220.ec2.internal\" not found" Jul 15 21:09:44.017133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.017084 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 21:04:43 +0000 UTC" deadline="2028-01-11 16:52:29.733578852 +0000 UTC" Jul 15 21:09:44.017133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.017131 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13075h42m45.716452939s" Jul 15 21:09:44.017308 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.017156 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d00899e62aada42f2f0bdc458167583d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal\" (UID: \"d00899e62aada42f2f0bdc458167583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.017308 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.017191 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d00899e62aada42f2f0bdc458167583d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal\" (UID: \"d00899e62aada42f2f0bdc458167583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.017308 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.017232 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45606745bd8d4c83782613a3e901378b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-220.ec2.internal\" (UID: \"45606745bd8d4c83782613a3e901378b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.020056 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.020035 2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:09:44.042778 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.042762 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.047413 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.047369 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.054812 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.054799 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:09:44.055620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.055607 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:09:44.113932 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.113910 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.118139 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.118123 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d00899e62aada42f2f0bdc458167583d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal\" (UID: \"d00899e62aada42f2f0bdc458167583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.118216 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.118147 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d00899e62aada42f2f0bdc458167583d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal\" (UID: \"d00899e62aada42f2f0bdc458167583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.118216 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.118165 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45606745bd8d4c83782613a3e901378b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-220.ec2.internal\" (UID: \"45606745bd8d4c83782613a3e901378b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.118216 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.118204 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/45606745bd8d4c83782613a3e901378b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-220.ec2.internal\" (UID: \"45606745bd8d4c83782613a3e901378b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.118339 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.118229 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d00899e62aada42f2f0bdc458167583d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal\" (UID: \"d00899e62aada42f2f0bdc458167583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.118339 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.118252 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d00899e62aada42f2f0bdc458167583d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal\" (UID: \"d00899e62aada42f2f0bdc458167583d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.125150 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.125134 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:09:44.125202 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.125172 2553 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.125202 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.125182 2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.135518 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.135499 2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:09:44.135605 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.135546 2553 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-220.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.232348 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.232305 2553 apiserver.go:52] "Watching apiserver" Jul 15 21:09:44.239723 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.239704 2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 21:09:44.240202 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.240183 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-w5bkr","kube-system/konnectivity-agent-2wxrz","openshift-cluster-node-tuning-operator/tuned-skxnr","openshift-dns/node-resolver-lqmdr","openshift-image-registry/node-ca-k957b","openshift-multus/multus-rr7c2","openshift-multus/network-metrics-daemon-8cnsf","kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal","openshift-multus/multus-additional-cni-plugins-vchqc","openshift-network-diagnostics/network-check-target-58zwt","openshift-network-operator/iptables-alerter-jdgz4","openshift-ovn-kubernetes/ovnkube-node-zqbh7"] Jul 15 21:09:44.241673 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.241656 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:44.241775 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.241755 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:09:44.242713 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.242699 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.243947 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.243929 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.245153 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.245134 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.245456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.245256 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:09:44.245456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.245383 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kx7c6\"" Jul 15 21:09:44.245635 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.245616 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 21:09:44.245703 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.245640 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 21:09:44.246520 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.246507 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.247126 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.247110 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 21:09:44.247197 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.247147 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:09:44.247235 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.247209 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zffpx\"" Jul 15 21:09:44.248206 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.248120 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.248299 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.248217 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 21:09:44.249820 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.248772 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 21:09:44.249820 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.249118 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m4nw6\"" Jul 15 21:09:44.249820 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.249686 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 21:09:44.250027 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.249954 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 21:09:44.250121 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.250102 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7xmwc\"" Jul 15 21:09:44.251049 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.250437 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 21:09:44.251049 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.250638 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 21:09:44.251418 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.251399 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jxgr8\"" Jul 15 21:09:44.252081 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.252059 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 21:09:44.252177 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.252066 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 21:09:44.252177 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.252067 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 21:09:44.252274 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.252182 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.253675 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.253659 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.253763 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.253715 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:44.254318 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.254303 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 21:09:44.254396 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.254326 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 21:09:44.254396 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.254326 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9qkls\"" Jul 15 21:09:44.254752 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.254736 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 21:09:44.255068 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.255052 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.256440 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.256422 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.257187 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.257172 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 21:09:44.257319 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.257303 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ph6wg\"" Jul 15 21:09:44.257370 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.257351 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 21:09:44.257691 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.257674 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.258655 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.258641 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 21:09:44.259073 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.259055 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:44.259157 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.259080 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 21:09:44.259157 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.259115 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9t7tk\"" Jul 15 21:09:44.259157 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.259127 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:44.259383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.259236 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 21:09:44.259383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.259293 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 21:09:44.260360 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.260344 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 21:09:44.260440 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.260375 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 21:09:44.260440 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.260419 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 21:09:44.260513 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.260442 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8rw9s\"" Jul 15 21:09:44.260513 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.260482 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 21:09:44.315580 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.315561 2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 21:09:44.320104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320088 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cnibin\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.320167 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320113 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.320167 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320132 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-socket-dir-parent\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.320167 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320149 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-tmp\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.320284 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320172 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-system-cni-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.320284 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320204 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47faff50-dced-41ee-b2da-a1ccb917bb9f-dbus\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.320284 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320247 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-var-lib-kubelet\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.320284 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320278 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-node-log\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.320435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320300 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-cni-bin\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.320435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320317 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbdv\" (UniqueName: \"kubernetes.io/projected/7d7cd9da-9939-42a9-9a3e-56129fc6f090-kube-api-access-dtbdv\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.320435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320334 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-system-cni-dir\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.320435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320349 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.320435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320365 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-run-ovn-kubernetes\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.320435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320405 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovnkube-config\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320448 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2f64\" (UniqueName: \"kubernetes.io/projected/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-kube-api-access-l2f64\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320471 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf2040dc-6d30-445b-b18b-ca448e201e15-iptables-alerter-script\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320489 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gds5f\" (UniqueName: \"kubernetes.io/projected/bf2040dc-6d30-445b-b18b-ca448e201e15-kube-api-access-gds5f\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320518 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-cnibin\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320547 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-hosts-file\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320569 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82ffd961-41ee-44d3-b38f-c74dafaa59ca-host\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320587 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320603 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.320657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320633 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-systemd-units\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320671 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82ffd961-41ee-44d3-b38f-c74dafaa59ca-serviceca\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320692 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-env-overrides\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320711 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320726 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320744 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9x8\" (UniqueName: \"kubernetes.io/projected/8dd406be-56be-4aa7-a84f-56fb3308fc02-kube-api-access-lm9x8\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320759 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-modprobe-d\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320807 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320829 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320843 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-systemd\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320876 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-etc-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320904 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-log-socket\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320929 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-os-release\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-cni-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.321053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.320981 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-k8s-cni-cncf-io\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321005 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-tuned\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321029 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-kubelet\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321058 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-os-release\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321081 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysconfig\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321106 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysctl-d\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321122 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2m75\" (UniqueName: \"kubernetes.io/projected/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-kube-api-access-x2m75\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321136 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovn-node-metrics-cert\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321164 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321187 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqdw8\" (UniqueName: \"kubernetes.io/projected/bb98c88c-8490-4747-ac99-546a6159a8d3-kube-api-access-pqdw8\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321200 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-netns\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321227 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-daemon-config\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321265 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-tmp-dir\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321289 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-ovn\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.321590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321329 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321366 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-cni-multus\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321393 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-conf-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321408 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysctl-conf\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-systemd\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321452 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321478 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovnkube-script-lib\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321493 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-kubelet\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321518 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hxw\" (UniqueName: \"kubernetes.io/projected/8ae616e3-61e9-41b2-b166-1790d9cba04d-kube-api-access-j9hxw\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321547 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cfec505-8d15-44c1-af49-2328d48f3a70-konnectivity-ca\") pod \"konnectivity-agent-2wxrz\" (UID: \"4cfec505-8d15-44c1-af49-2328d48f3a70\") " pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321577 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321617 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cni-binary-copy\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321647 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf2040dc-6d30-445b-b18b-ca448e201e15-host-slash\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.322104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321670 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dd406be-56be-4aa7-a84f-56fb3308fc02-cni-binary-copy\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321695 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47faff50-dced-41ee-b2da-a1ccb917bb9f-kubelet-config\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321718 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-run\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321741 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-slash\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321768 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-hostroot\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321817 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-multus-certs\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321866 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-lib-modules\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321894 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-host\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321909 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cfec505-8d15-44c1-af49-2328d48f3a70-agent-certs\") pod \"konnectivity-agent-2wxrz\" (UID: \"4cfec505-8d15-44c1-af49-2328d48f3a70\") " pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321923 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-var-lib-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321939 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321962 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-sys\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.321994 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-device-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.322010 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-run-netns\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.322455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.322024 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-cni-netd\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.322877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.322046 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldxsj\" (UniqueName: \"kubernetes.io/projected/82ffd961-41ee-44d3-b38f-c74dafaa59ca-kube-api-access-ldxsj\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.322877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.322060 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-cni-bin\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.322074 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-etc-kubernetes\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.322877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.322088 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-kubernetes\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.322877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.322106 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtqw\" (UniqueName: \"kubernetes.io/projected/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-kube-api-access-ngtqw\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.357995 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.357973 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.358837 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.358806 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" Jul 15 21:09:44.422641 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422604 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-kubernetes\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.422819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtqw\" (UniqueName: \"kubernetes.io/projected/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-kube-api-access-ngtqw\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.422819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422671 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cnibin\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.422819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422717 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-kubernetes\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.422819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422723 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cnibin\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422814 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422847 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-socket-dir-parent\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422865 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-tmp\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422882 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-system-cni-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422906 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47faff50-dced-41ee-b2da-a1ccb917bb9f-dbus\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422927 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-var-lib-kubelet\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422982 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-node-log\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.423009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.422990 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-var-lib-kubelet\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423017 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-cni-bin\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423037 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-socket-dir-parent\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423044 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbdv\" (UniqueName: \"kubernetes.io/projected/7d7cd9da-9939-42a9-9a3e-56129fc6f090-kube-api-access-dtbdv\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423090 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-system-cni-dir\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423105 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-node-log\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423117 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/47faff50-dced-41ee-b2da-a1ccb917bb9f-dbus\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423125 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423141 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-cni-bin\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423154 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-run-ovn-kubernetes\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423160 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-system-cni-dir\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423183 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovnkube-config\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423214 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2f64\" (UniqueName: \"kubernetes.io/projected/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-kube-api-access-l2f64\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-registration-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.423383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423242 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf2040dc-6d30-445b-b18b-ca448e201e15-iptables-alerter-script\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423269 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gds5f\" (UniqueName: \"kubernetes.io/projected/bf2040dc-6d30-445b-b18b-ca448e201e15-kube-api-access-gds5f\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423273 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-run-ovn-kubernetes\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423272 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-system-cni-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423275 2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423293 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-cnibin\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423362 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-cnibin\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423425 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-hosts-file\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423456 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82ffd961-41ee-44d3-b38f-c74dafaa59ca-host\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423481 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423492 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423507 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423509 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-hosts-file\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423530 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-systemd-units\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423554 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82ffd961-41ee-44d3-b38f-c74dafaa59ca-serviceca\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.424152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423577 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-env-overrides\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423602 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423613 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/82ffd961-41ee-44d3-b38f-c74dafaa59ca-host\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423626 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423657 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9x8\" (UniqueName: \"kubernetes.io/projected/8dd406be-56be-4aa7-a84f-56fb3308fc02-kube-api-access-lm9x8\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423682 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-modprobe-d\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423683 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-sys-fs\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423721 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423746 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423771 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-systemd\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423794 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-etc-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423818 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-log-socket\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423858 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-os-release\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423862 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.424930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423883 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-cni-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423911 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-k8s-cni-cncf-io\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423915 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovnkube-config\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423936 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-tuned\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423962 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-kubelet\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423986 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-os-release\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-etc-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424010 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysconfig\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424035 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysctl-d\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424037 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-env-overrides\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.423624 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-socket-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424087 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-modprobe-d\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424102 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-log-socket\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424111 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-k8s-cni-cncf-io\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424129 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/82ffd961-41ee-44d3-b38f-c74dafaa59ca-serviceca\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.425676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424123 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bf2040dc-6d30-445b-b18b-ca448e201e15-iptables-alerter-script\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424145 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424174 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-cni-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424173 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-os-release\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424188 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-systemd\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-systemd-units\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424060 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2m75\" (UniqueName: \"kubernetes.io/projected/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-kube-api-access-x2m75\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424230 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-kubelet\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424246 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovn-node-metrics-cert\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424276 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424294 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysctl-d\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqdw8\" (UniqueName: \"kubernetes.io/projected/bb98c88c-8490-4747-ac99-546a6159a8d3-kube-api-access-pqdw8\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424293 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424358 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-netns\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.424382 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:44.426515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424419 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-os-release\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424463 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-netns\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.424477 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret podName:47faff50-dced-41ee-b2da-a1ccb917bb9f nodeName:}" failed. No retries permitted until 2026-07-15 21:09:44.924435721 +0000 UTC m=+3.051099690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret") pod "global-pull-secret-syncer-w5bkr" (UID: "47faff50-dced-41ee-b2da-a1ccb917bb9f") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424496 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-daemon-config\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424523 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-tmp-dir\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424547 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-ovn\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424572 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424627 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysconfig\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424663 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-cni-multus\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424690 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-conf-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424712 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysctl-conf\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424735 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-systemd\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424803 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovnkube-script-lib\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.427168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424828 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-kubelet\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hxw\" (UniqueName: \"kubernetes.io/projected/8ae616e3-61e9-41b2-b166-1790d9cba04d-kube-api-access-j9hxw\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-ovn\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424881 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cfec505-8d15-44c1-af49-2328d48f3a70-konnectivity-ca\") pod \"konnectivity-agent-2wxrz\" (UID: \"4cfec505-8d15-44c1-af49-2328d48f3a70\") " pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424903 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-tmp-dir\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424972 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-systemd\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.424984 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-conf-dir\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425010 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-cni-multus\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425060 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-kubelet\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425060 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425087 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8dd406be-56be-4aa7-a84f-56fb3308fc02-multus-daemon-config\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425096 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cni-binary-copy\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425126 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425139 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf2040dc-6d30-445b-b18b-ca448e201e15-host-slash\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.427917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425158 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425180 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bf2040dc-6d30-445b-b18b-ca448e201e15-host-slash\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425179 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dd406be-56be-4aa7-a84f-56fb3308fc02-cni-binary-copy\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425220 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47faff50-dced-41ee-b2da-a1ccb917bb9f-kubelet-config\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425247 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-run\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425273 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-slash\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425298 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-hostroot\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425319 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-run-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425323 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-multus-certs\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425361 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-run-multus-certs\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425373 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-lib-modules\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425397 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-host\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425411 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/47faff50-dced-41ee-b2da-a1ccb917bb9f-kubelet-config\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425421 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cfec505-8d15-44c1-af49-2328d48f3a70-agent-certs\") pod \"konnectivity-agent-2wxrz\" (UID: \"4cfec505-8d15-44c1-af49-2328d48f3a70\") " pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-var-lib-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425462 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-run\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425475 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-sys\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425525 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-device-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425530 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-hostroot\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425543 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovnkube-script-lib\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425566 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-run-netns\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425590 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-cni-netd\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425614 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldxsj\" (UniqueName: \"kubernetes.io/projected/82ffd961-41ee-44d3-b38f-c74dafaa59ca-kube-api-access-ldxsj\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425635 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-cni-bin\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425646 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dd406be-56be-4aa7-a84f-56fb3308fc02-cni-binary-copy\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425657 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-etc-kubernetes\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425501 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-slash\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425709 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-cni-binary-copy\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425740 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-cni-netd\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.428751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425741 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-host-run-netns\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425781 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-host-var-lib-cni-bin\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425815 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-host\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425862 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-lib-modules\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.425992 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8dd406be-56be-4aa7-a84f-56fb3308fc02-etc-kubernetes\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.426007 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4cfec505-8d15-44c1-af49-2328d48f3a70-konnectivity-ca\") pod \"konnectivity-agent-2wxrz\" (UID: \"4cfec505-8d15-44c1-af49-2328d48f3a70\") " pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.426037 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8ae616e3-61e9-41b2-b166-1790d9cba04d-device-dir\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.426044 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-sys\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.426076 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d7cd9da-9939-42a9-9a3e-56129fc6f090-var-lib-openvswitch\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.426156 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.426238 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-sysctl-conf\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.426266 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs podName:bb98c88c-8490-4747-ac99-546a6159a8d3 nodeName:}" failed. No retries permitted until 2026-07-15 21:09:44.926247914 +0000 UTC m=+3.052911890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs") pod "network-metrics-daemon-8cnsf" (UID: "bb98c88c-8490-4747-ac99-546a6159a8d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.426971 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-tmp\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.427282 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d7cd9da-9939-42a9-9a3e-56129fc6f090-ovn-node-metrics-cert\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.427449 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-etc-tuned\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.429201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.428061 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4cfec505-8d15-44c1-af49-2328d48f3a70-agent-certs\") pod \"konnectivity-agent-2wxrz\" (UID: \"4cfec505-8d15-44c1-af49-2328d48f3a70\") " pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.430215 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.430195 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:09:44.430293 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.430222 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:09:44.430293 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.430236 2553 projected.go:196] Error preparing data for projected volume kube-api-access-5hfqs for pod openshift-network-diagnostics/network-check-target-58zwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:44.430395 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.430308 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs podName:ce4f6bd7-9e54-4730-9afc-5d1f14ed529e nodeName:}" failed. No retries permitted until 2026-07-15 21:09:44.930291787 +0000 UTC m=+3.056955771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5hfqs" (UniqueName: "kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs") pod "network-check-target-58zwt" (UID: "ce4f6bd7-9e54-4730-9afc-5d1f14ed529e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:44.432467 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.432392 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2f64\" (UniqueName: \"kubernetes.io/projected/ca0d0d1d-4805-4ad6-9b4a-38b0e621affc-kube-api-access-l2f64\") pod \"multus-additional-cni-plugins-vchqc\" (UID: \"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc\") " pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.432574 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.432508 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gds5f\" (UniqueName: \"kubernetes.io/projected/bf2040dc-6d30-445b-b18b-ca448e201e15-kube-api-access-gds5f\") pod \"iptables-alerter-jdgz4\" (UID: \"bf2040dc-6d30-445b-b18b-ca448e201e15\") " pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.432826 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.432799 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9x8\" (UniqueName: \"kubernetes.io/projected/8dd406be-56be-4aa7-a84f-56fb3308fc02-kube-api-access-lm9x8\") pod \"multus-rr7c2\" (UID: \"8dd406be-56be-4aa7-a84f-56fb3308fc02\") " pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.434048 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.434025 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqdw8\" (UniqueName: \"kubernetes.io/projected/bb98c88c-8490-4747-ac99-546a6159a8d3-kube-api-access-pqdw8\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:44.434561 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.434527 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtqw\" (UniqueName: \"kubernetes.io/projected/bc1abd10-3d1b-4792-8a45-93a7d614d7e7-kube-api-access-ngtqw\") pod \"tuned-skxnr\" (UID: \"bc1abd10-3d1b-4792-8a45-93a7d614d7e7\") " pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.434743 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.434720 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbdv\" (UniqueName: \"kubernetes.io/projected/7d7cd9da-9939-42a9-9a3e-56129fc6f090-kube-api-access-dtbdv\") pod \"ovnkube-node-zqbh7\" (UID: \"7d7cd9da-9939-42a9-9a3e-56129fc6f090\") " pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.434827 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.434803 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hxw\" (UniqueName: \"kubernetes.io/projected/8ae616e3-61e9-41b2-b166-1790d9cba04d-kube-api-access-j9hxw\") pod \"aws-ebs-csi-driver-node-8dtkt\" (UID: \"8ae616e3-61e9-41b2-b166-1790d9cba04d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.435229 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.435210 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldxsj\" (UniqueName: \"kubernetes.io/projected/82ffd961-41ee-44d3-b38f-c74dafaa59ca-kube-api-access-ldxsj\") pod \"node-ca-k957b\" (UID: \"82ffd961-41ee-44d3-b38f-c74dafaa59ca\") " pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.435894 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.435870 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2m75\" (UniqueName: \"kubernetes.io/projected/00459e80-47cb-4df5-a7d0-4a6c6d0a173c-kube-api-access-x2m75\") pod \"node-resolver-lqmdr\" (UID: \"00459e80-47cb-4df5-a7d0-4a6c6d0a173c\") " pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.491850 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.491829 2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:09:44.532058 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.532040 2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:09:44.567885 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.567833 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdgz4" Jul 15 21:09:44.582893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.582877 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-skxnr" Jul 15 21:09:44.595721 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.595702 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lqmdr" Jul 15 21:09:44.610447 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.610422 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k957b" Jul 15 21:09:44.628614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.628590 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rr7c2" Jul 15 21:09:44.645933 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.645914 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" Jul 15 21:09:44.659868 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.659851 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:09:44.670731 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.670716 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:09:44.675343 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.675326 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vchqc" Jul 15 21:09:44.897439 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:44.897396 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45606745bd8d4c83782613a3e901378b.slice/crio-635c0e7fe1dc46445b784e3e8b36e9ec6e42ec2fb3ac663485813715f5327a31 WatchSource:0}: Error finding container 635c0e7fe1dc46445b784e3e8b36e9ec6e42ec2fb3ac663485813715f5327a31: Status 404 returned error can't find the container with id 635c0e7fe1dc46445b784e3e8b36e9ec6e42ec2fb3ac663485813715f5327a31 Jul 15 21:09:44.897743 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:44.897722 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00899e62aada42f2f0bdc458167583d.slice/crio-8c00681faf347a75ce35ca645b14239782e30e8adfa11094c6eec53e4a1c1b09 WatchSource:0}: Error finding container 8c00681faf347a75ce35ca645b14239782e30e8adfa11094c6eec53e4a1c1b09: Status 404 returned error can't find the container with id 8c00681faf347a75ce35ca645b14239782e30e8adfa11094c6eec53e4a1c1b09 Jul 15 21:09:44.903612 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.903594 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:09:44.928798 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.928760 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:44.928907 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:44.928836 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:44.928907 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.928883 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:44.929017 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.928931 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:44.929017 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.928970 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs podName:bb98c88c-8490-4747-ac99-546a6159a8d3 nodeName:}" failed. No retries permitted until 2026-07-15 21:09:45.92894588 +0000 UTC m=+4.055609841 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs") pod "network-metrics-daemon-8cnsf" (UID: "bb98c88c-8490-4747-ac99-546a6159a8d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:44.929017 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:44.928992 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret podName:47faff50-dced-41ee-b2da-a1ccb917bb9f nodeName:}" failed. No retries permitted until 2026-07-15 21:09:45.928980691 +0000 UTC m=+4.055644652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret") pod "global-pull-secret-syncer-w5bkr" (UID: "47faff50-dced-41ee-b2da-a1ccb917bb9f") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:44.944647 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:44.944620 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00459e80_47cb_4df5_a7d0_4a6c6d0a173c.slice/crio-30da71b9c4bc92a35483ca70f32d9c2e53820497ad9f739a2fb2f30b0f6ec102 WatchSource:0}: Error finding container 30da71b9c4bc92a35483ca70f32d9c2e53820497ad9f739a2fb2f30b0f6ec102: Status 404 returned error can't find the container with id 30da71b9c4bc92a35483ca70f32d9c2e53820497ad9f739a2fb2f30b0f6ec102 Jul 15 21:09:44.945018 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:44.945000 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd406be_56be_4aa7_a84f_56fb3308fc02.slice/crio-c98a28d2f4f6d4b55f49236bb4e478925f3f9fc4d0fd267fec485f603eb85eca WatchSource:0}: Error finding container c98a28d2f4f6d4b55f49236bb4e478925f3f9fc4d0fd267fec485f603eb85eca: Status 404 returned error can't find the container with id c98a28d2f4f6d4b55f49236bb4e478925f3f9fc4d0fd267fec485f603eb85eca Jul 15 21:09:44.965536 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:44.965519 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfec505_8d15_44c1_af49_2328d48f3a70.slice/crio-835e03dd70d2a98cc1db6dd9541ae40e51db09d7f54dbfaa87f15188d05b6f78 WatchSource:0}: Error finding container 835e03dd70d2a98cc1db6dd9541ae40e51db09d7f54dbfaa87f15188d05b6f78: Status 404 returned error can't find the container with id 835e03dd70d2a98cc1db6dd9541ae40e51db09d7f54dbfaa87f15188d05b6f78 Jul 15 21:09:45.018026 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.018001 2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 21:09:45.022699 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:45.022675 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2040dc_6d30_445b_b18b_ca448e201e15.slice/crio-5629de72cd1427391d9ef6b7c9b08935550b610abd5915ecc7b9bd014c272513 WatchSource:0}: Error finding container 5629de72cd1427391d9ef6b7c9b08935550b610abd5915ecc7b9bd014c272513: Status 404 returned error can't find the container with id 5629de72cd1427391d9ef6b7c9b08935550b610abd5915ecc7b9bd014c272513 Jul 15 21:09:45.028897 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.028878 2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 21:09:45.029144 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.029121 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:45.029265 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.029253 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:09:45.029313 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.029271 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:09:45.029313 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.029283 2553 projected.go:196] Error preparing data for projected volume kube-api-access-5hfqs for pod openshift-network-diagnostics/network-check-target-58zwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:45.029390 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.029331 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs podName:ce4f6bd7-9e54-4730-9afc-5d1f14ed529e nodeName:}" failed. No retries permitted until 2026-07-15 21:09:46.029317994 +0000 UTC m=+4.155981955 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hfqs" (UniqueName: "kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs") pod "network-check-target-58zwt" (UID: "ce4f6bd7-9e54-4730-9afc-5d1f14ed529e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:45.034719 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:45.034700 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0d0d1d_4805_4ad6_9b4a_38b0e621affc.slice/crio-e0fd437423f61d54923449906ea87d8881f0fcdfe5ff0a374d06062805b01ade WatchSource:0}: Error finding container e0fd437423f61d54923449906ea87d8881f0fcdfe5ff0a374d06062805b01ade: Status 404 returned error can't find the container with id e0fd437423f61d54923449906ea87d8881f0fcdfe5ff0a374d06062805b01ade Jul 15 21:09:45.045827 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:45.045793 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae616e3_61e9_41b2_b166_1790d9cba04d.slice/crio-b572a34a613254416f2656427b923b94d8cd3161f2195f1fbc239b9a52dbc703 WatchSource:0}: Error finding container b572a34a613254416f2656427b923b94d8cd3161f2195f1fbc239b9a52dbc703: Status 404 returned error can't find the container with id b572a34a613254416f2656427b923b94d8cd3161f2195f1fbc239b9a52dbc703 Jul 15 21:09:45.058342 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.058280 2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5hhfg" Jul 15 21:09:45.061052 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:45.061019 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1abd10_3d1b_4792_8a45_93a7d614d7e7.slice/crio-440bf5e832567f7a0d7be219f1bbfa8486c4d33e48b095cd202659b3fadca992 WatchSource:0}: Error finding container 440bf5e832567f7a0d7be219f1bbfa8486c4d33e48b095cd202659b3fadca992: Status 404 returned error can't find the container with id 440bf5e832567f7a0d7be219f1bbfa8486c4d33e48b095cd202659b3fadca992 Jul 15 21:09:45.064799 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.064783 2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5hhfg" Jul 15 21:09:45.144293 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.144212 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-skxnr" event={"ID":"bc1abd10-3d1b-4792-8a45-93a7d614d7e7","Type":"ContainerStarted","Data":"440bf5e832567f7a0d7be219f1bbfa8486c4d33e48b095cd202659b3fadca992"} Jul 15 21:09:45.145218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.145195 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" event={"ID":"8ae616e3-61e9-41b2-b166-1790d9cba04d","Type":"ContainerStarted","Data":"b572a34a613254416f2656427b923b94d8cd3161f2195f1fbc239b9a52dbc703"} Jul 15 21:09:45.146191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.146172 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdgz4" event={"ID":"bf2040dc-6d30-445b-b18b-ca448e201e15","Type":"ContainerStarted","Data":"5629de72cd1427391d9ef6b7c9b08935550b610abd5915ecc7b9bd014c272513"} Jul 15 21:09:45.147258 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.147221 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lqmdr" event={"ID":"00459e80-47cb-4df5-a7d0-4a6c6d0a173c","Type":"ContainerStarted","Data":"30da71b9c4bc92a35483ca70f32d9c2e53820497ad9f739a2fb2f30b0f6ec102"} Jul 15 21:09:45.148287 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.148262 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" event={"ID":"d00899e62aada42f2f0bdc458167583d","Type":"ContainerStarted","Data":"8c00681faf347a75ce35ca645b14239782e30e8adfa11094c6eec53e4a1c1b09"} Jul 15 21:09:45.149308 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.149291 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerStarted","Data":"e0fd437423f61d54923449906ea87d8881f0fcdfe5ff0a374d06062805b01ade"} Jul 15 21:09:45.150344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.150324 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wxrz" event={"ID":"4cfec505-8d15-44c1-af49-2328d48f3a70","Type":"ContainerStarted","Data":"835e03dd70d2a98cc1db6dd9541ae40e51db09d7f54dbfaa87f15188d05b6f78"} Jul 15 21:09:45.151332 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.151314 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rr7c2" event={"ID":"8dd406be-56be-4aa7-a84f-56fb3308fc02","Type":"ContainerStarted","Data":"c98a28d2f4f6d4b55f49236bb4e478925f3f9fc4d0fd267fec485f603eb85eca"} Jul 15 21:09:45.152256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.152233 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" event={"ID":"45606745bd8d4c83782613a3e901378b","Type":"ContainerStarted","Data":"635c0e7fe1dc46445b784e3e8b36e9ec6e42ec2fb3ac663485813715f5327a31"} Jul 15 21:09:45.230584 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:45.230563 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ffd961_41ee_44d3_b38f_c74dafaa59ca.slice/crio-9218e609e12b3a9fadb305e81bfe1911b48fe5b7b7bcfc549c23510e58afdf11 WatchSource:0}: Error finding container 9218e609e12b3a9fadb305e81bfe1911b48fe5b7b7bcfc549c23510e58afdf11: Status 404 returned error can't find the container with id 9218e609e12b3a9fadb305e81bfe1911b48fe5b7b7bcfc549c23510e58afdf11 Jul 15 21:09:45.408493 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:09:45.408177 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7cd9da_9939_42a9_9a3e_56129fc6f090.slice/crio-9575ae5840ab8b104a60910a46e51a8fafcee52676e4b46108ddb67d6f5ded35 WatchSource:0}: Error finding container 9575ae5840ab8b104a60910a46e51a8fafcee52676e4b46108ddb67d6f5ded35: Status 404 returned error can't find the container with id 9575ae5840ab8b104a60910a46e51a8fafcee52676e4b46108ddb67d6f5ded35 Jul 15 21:09:45.936300 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.936265 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:45.936467 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:45.936360 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:45.936542 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.936487 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:45.936653 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.936549 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret podName:47faff50-dced-41ee-b2da-a1ccb917bb9f nodeName:}" failed. No retries permitted until 2026-07-15 21:09:47.936528878 +0000 UTC m=+6.063192846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret") pod "global-pull-secret-syncer-w5bkr" (UID: "47faff50-dced-41ee-b2da-a1ccb917bb9f") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:45.936946 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.936931 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:45.937022 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:45.937003 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs podName:bb98c88c-8490-4747-ac99-546a6159a8d3 nodeName:}" failed. No retries permitted until 2026-07-15 21:09:47.936973753 +0000 UTC m=+6.063637716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs") pod "network-metrics-daemon-8cnsf" (UID: "bb98c88c-8490-4747-ac99-546a6159a8d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:46.037325 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.036917 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:46.037325 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:46.037120 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:09:46.037325 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:46.037139 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:09:46.037325 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:46.037152 2553 projected.go:196] Error preparing data for projected volume kube-api-access-5hfqs for pod openshift-network-diagnostics/network-check-target-58zwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:46.037325 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:46.037205 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs podName:ce4f6bd7-9e54-4730-9afc-5d1f14ed529e nodeName:}" failed. No retries permitted until 2026-07-15 21:09:48.037186234 +0000 UTC m=+6.163850197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hfqs" (UniqueName: "kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs") pod "network-check-target-58zwt" (UID: "ce4f6bd7-9e54-4730-9afc-5d1f14ed529e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:46.066860 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.066815 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 21:04:45 +0000 UTC" deadline="2028-04-20 12:54:51.10471245 +0000 UTC" Jul 15 21:09:46.066860 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.066860 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15471h45m5.037857578s" Jul 15 21:09:46.137744 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.137705 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:46.137928 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:46.137868 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:09:46.138005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.137943 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:46.138110 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:46.138059 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:46.138168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.138135 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:46.138223 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:46.138206 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:46.175520 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.175490 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"9575ae5840ab8b104a60910a46e51a8fafcee52676e4b46108ddb67d6f5ded35"} Jul 15 21:09:46.192677 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:46.192613 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k957b" event={"ID":"82ffd961-41ee-44d3-b38f-c74dafaa59ca","Type":"ContainerStarted","Data":"9218e609e12b3a9fadb305e81bfe1911b48fe5b7b7bcfc549c23510e58afdf11"} Jul 15 21:09:47.067067 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:47.066973 2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 21:04:45 +0000 UTC" deadline="2028-04-25 03:39:58.608719556 +0000 UTC" Jul 15 21:09:47.067067 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:47.067017 2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15582h30m11.541706612s" Jul 15 21:09:47.954256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:47.954135 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:47.954256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:47.954220 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:47.954530 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:47.954373 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:47.954530 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:47.954438 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs podName:bb98c88c-8490-4747-ac99-546a6159a8d3 nodeName:}" failed. No retries permitted until 2026-07-15 21:09:51.954417973 +0000 UTC m=+10.081081934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs") pod "network-metrics-daemon-8cnsf" (UID: "bb98c88c-8490-4747-ac99-546a6159a8d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:47.954956 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:47.954889 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:47.954956 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:47.954943 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret podName:47faff50-dced-41ee-b2da-a1ccb917bb9f nodeName:}" failed. No retries permitted until 2026-07-15 21:09:51.954928118 +0000 UTC m=+10.081592082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret") pod "global-pull-secret-syncer-w5bkr" (UID: "47faff50-dced-41ee-b2da-a1ccb917bb9f") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:48.054925 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:48.054891 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:48.055089 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:48.055070 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:09:48.055162 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:48.055091 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:09:48.055162 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:48.055104 2553 projected.go:196] Error preparing data for projected volume kube-api-access-5hfqs for pod openshift-network-diagnostics/network-check-target-58zwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:48.055313 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:48.055164 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs podName:ce4f6bd7-9e54-4730-9afc-5d1f14ed529e nodeName:}" failed. No retries permitted until 2026-07-15 21:09:52.055144818 +0000 UTC m=+10.181808794 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hfqs" (UniqueName: "kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs") pod "network-check-target-58zwt" (UID: "ce4f6bd7-9e54-4730-9afc-5d1f14ed529e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:48.138629 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:48.138600 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:48.139160 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:48.138735 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:09:48.139160 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:48.138844 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:48.139160 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:48.138925 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:48.139160 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:48.138994 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:48.139160 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:48.139063 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:50.138023 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:50.137994 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:50.138599 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:50.138125 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:50.138599 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:50.138295 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:09:50.138599 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:50.138346 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:50.138599 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:50.138419 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:50.138599 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:50.138120 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:51.989575 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:51.989535 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:51.990129 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:51.989624 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:51.990129 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:51.989716 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:51.990129 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:51.989751 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:51.990129 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:51.989830 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret podName:47faff50-dced-41ee-b2da-a1ccb917bb9f nodeName:}" failed. No retries permitted until 2026-07-15 21:09:59.989787928 +0000 UTC m=+18.116451887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret") pod "global-pull-secret-syncer-w5bkr" (UID: "47faff50-dced-41ee-b2da-a1ccb917bb9f") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:09:51.990129 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:51.989853 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs podName:bb98c88c-8490-4747-ac99-546a6159a8d3 nodeName:}" failed. No retries permitted until 2026-07-15 21:09:59.989841321 +0000 UTC m=+18.116505283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs") pod "network-metrics-daemon-8cnsf" (UID: "bb98c88c-8490-4747-ac99-546a6159a8d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:09:52.091178 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:52.090707 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:52.091178 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:52.090903 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:09:52.091178 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:52.090924 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:09:52.091178 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:52.090937 2553 projected.go:196] Error preparing data for projected volume kube-api-access-5hfqs for pod openshift-network-diagnostics/network-check-target-58zwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:52.091178 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:52.090997 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs podName:ce4f6bd7-9e54-4730-9afc-5d1f14ed529e nodeName:}" failed. No retries permitted until 2026-07-15 21:10:00.09097651 +0000 UTC m=+18.217640472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hfqs" (UniqueName: "kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs") pod "network-check-target-58zwt" (UID: "ce4f6bd7-9e54-4730-9afc-5d1f14ed529e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:09:52.138608 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:52.138063 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:52.138608 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:52.138113 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:52.138608 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:52.138203 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:52.138608 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:52.138542 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:52.138957 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:52.138648 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:09:52.138957 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:52.138747 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:54.138170 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:54.137840 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:54.138170 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:54.137893 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:54.138170 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:54.137966 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:09:54.138170 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:54.137844 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:54.138170 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:54.138073 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:54.138170 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:54.138167 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:56.137888 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:56.137843 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:56.137888 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:56.137867 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:56.137888 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:56.137867 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:56.138371 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:56.137982 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:56.138371 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:56.138043 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:09:56.138371 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:56.138130 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:58.137943 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:58.137909 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:09:58.138342 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:58.138024 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:09:58.138342 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:58.138038 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:09:58.138342 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:09:58.138061 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:09:58.138342 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:58.138140 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:09:58.138342 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:09:58.138244 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:00.051039 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:00.051002 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:00.051448 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:00.051059 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:00.051448 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.051190 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:00.051448 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.051265 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret podName:47faff50-dced-41ee-b2da-a1ccb917bb9f nodeName:}" failed. No retries permitted until 2026-07-15 21:10:16.051243858 +0000 UTC m=+34.177907822 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret") pod "global-pull-secret-syncer-w5bkr" (UID: "47faff50-dced-41ee-b2da-a1ccb917bb9f") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:00.051448 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.051193 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:00.051448 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.051347 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs podName:bb98c88c-8490-4747-ac99-546a6159a8d3 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:16.051329459 +0000 UTC m=+34.177993432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs") pod "network-metrics-daemon-8cnsf" (UID: "bb98c88c-8490-4747-ac99-546a6159a8d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:00.137705 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:00.137672 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:00.137896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:00.137717 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:00.137896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:00.137669 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:00.137896 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.137801 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:00.138050 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.137907 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:00.138050 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.137965 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:00.152178 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:00.152156 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:00.152322 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.152305 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:00.152382 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.152323 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:00.152382 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.152335 2553 projected.go:196] Error preparing data for projected volume kube-api-access-5hfqs for pod openshift-network-diagnostics/network-check-target-58zwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:00.152464 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:00.152388 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs podName:ce4f6bd7-9e54-4730-9afc-5d1f14ed529e nodeName:}" failed. No retries permitted until 2026-07-15 21:10:16.152370465 +0000 UTC m=+34.279034424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hfqs" (UniqueName: "kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs") pod "network-check-target-58zwt" (UID: "ce4f6bd7-9e54-4730-9afc-5d1f14ed529e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:02.138292 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:02.138239 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:02.138292 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:02.138262 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:02.138292 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:02.138280 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:02.138855 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:02.138379 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:02.138855 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:02.138447 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:02.138855 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:02.138503 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:04.137786 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:04.137747 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:04.138236 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:04.137788 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:04.138236 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:04.137797 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:04.138236 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:04.137881 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:04.138236 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:04.137986 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:04.138236 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:04.138065 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:05.237273 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:05.237045 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rr7c2" event={"ID":"8dd406be-56be-4aa7-a84f-56fb3308fc02","Type":"ContainerStarted","Data":"b7fb3ef6b295195de3d0fe10ca579d866d7f1d39d2197d38dc876b5da7a39e9a"} Jul 15 21:10:05.238622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:05.238604 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" event={"ID":"45606745bd8d4c83782613a3e901378b","Type":"ContainerStarted","Data":"9cb297fc10fe56def13368a33cb6137f6210cd3a32f80353b4abcf6ea137eea0"} Jul 15 21:10:05.240212 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:05.240191 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-skxnr" event={"ID":"bc1abd10-3d1b-4792-8a45-93a7d614d7e7","Type":"ContainerStarted","Data":"8bb4821e6e7ed242da70b5bd00e0dcdbde890ee3915b49fbd58de07a6568a948"} Jul 15 21:10:05.291308 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:05.291263 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rr7c2" podStartSLOduration=2.330494617 podStartE2EDuration="22.291247871s" totalImagesPullingTime="19.960753254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:05.271173001 +0000 UTC m=+23.397836982" watchObservedRunningTime="2026-07-15 21:10:05.291247871 +0000 UTC m=+23.417911846" Jul 15 21:10:05.291424 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:05.291399 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-220.ec2.internal" podStartSLOduration=21.291395405 podStartE2EDuration="21.291395405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:05.290890292 +0000 UTC m=+23.417554272" watchObservedRunningTime="2026-07-15 21:10:05.291395405 +0000 UTC m=+23.418059380" Jul 15 21:10:05.319219 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:05.319153 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-skxnr" podStartSLOduration=2.475903373 podStartE2EDuration="22.319144467s" totalImagesPullingTime="19.843241094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:05.31892716 +0000 UTC m=+23.445591134" watchObservedRunningTime="2026-07-15 21:10:05.319144467 +0000 UTC m=+23.445808441" Jul 15 21:10:06.138620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.138471 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:06.138722 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.138532 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:06.138722 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:06.138693 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:06.138856 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.138537 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:06.138856 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:06.138782 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:06.138939 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:06.138874 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:06.242734 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.242710 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k957b" event={"ID":"82ffd961-41ee-44d3-b38f-c74dafaa59ca","Type":"ContainerStarted","Data":"19021cef2e956e7b24b73241dcc1af44c2bfd2ca131879de3d8113e44ddbc8dc"} Jul 15 21:10:06.244089 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.244058 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" event={"ID":"8ae616e3-61e9-41b2-b166-1790d9cba04d","Type":"ContainerStarted","Data":"27bbdd4d6c99e3f2de775efe1eebaf7d53ad499c884865cfab6777f6ffd2fa76"} Jul 15 21:10:06.245345 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.245327 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdgz4" event={"ID":"bf2040dc-6d30-445b-b18b-ca448e201e15","Type":"ContainerStarted","Data":"6e5d0ae690107e48e626749c248f93272f2087dbac1a7de1bbcf39c15844f666"} Jul 15 21:10:06.246598 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.246580 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lqmdr" event={"ID":"00459e80-47cb-4df5-a7d0-4a6c6d0a173c","Type":"ContainerStarted","Data":"09fc53a071f50fa1e1a50b355d7de408d56feffac6cf4c78f6397eaaa26c36f5"} Jul 15 21:10:06.247979 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.247960 2553 generic.go:356] "Generic (PLEG): container finished" podID="d00899e62aada42f2f0bdc458167583d" containerID="403457baa3e8928b659d5d6c79ebb0d958f8752c929142a8f5ed40cf0bfcb3da" exitCode=0 Jul 15 21:10:06.248077 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.248037 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" event={"ID":"d00899e62aada42f2f0bdc458167583d","Type":"ContainerDied","Data":"403457baa3e8928b659d5d6c79ebb0d958f8752c929142a8f5ed40cf0bfcb3da"} Jul 15 21:10:06.250629 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.250614 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:10:06.250973 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.250956 2553 generic.go:356] "Generic (PLEG): container finished" podID="7d7cd9da-9939-42a9-9a3e-56129fc6f090" containerID="fd5c6746861930e13ba27eafcaeb27c3f901e8f7f638fc11dbbe88e5e1930e40" exitCode=1 Jul 15 21:10:06.251029 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.251013 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"c79e33e95f8e16d687e2ffe7e61130c9c9d504772646d1bd16fb6257624a839d"} Jul 15 21:10:06.251099 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.251078 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"c2cc27b413cd02987ab566a4c529d4d11622bfefb5598c13570db81e08d653da"} Jul 15 21:10:06.251146 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.251133 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"d8a04a16d4d76dd6d74eaafeeb8fe7c114386e36e999448a033d815f6521b33c"} Jul 15 21:10:06.251193 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.251181 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"6d6800c2d830edad502e7ff9a734a3a6a5c07f6666228563527b45aff6bb338e"} Jul 15 21:10:06.251241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.251229 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerDied","Data":"fd5c6746861930e13ba27eafcaeb27c3f901e8f7f638fc11dbbe88e5e1930e40"} Jul 15 21:10:06.251280 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.251272 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"cb20d2f11af939844e6fbe7fe0e0ad4824fe69c324042f6da55b54e9e18fd871"} Jul 15 21:10:06.254846 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.254823 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca0d0d1d-4805-4ad6-9b4a-38b0e621affc" containerID="3fdb391f4c4502cfebc1ac63be4c3a83cea4bd4edb8993bb6b760e520ef13e2c" exitCode=0 Jul 15 21:10:06.254939 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.254910 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerDied","Data":"3fdb391f4c4502cfebc1ac63be4c3a83cea4bd4edb8993bb6b760e520ef13e2c"} Jul 15 21:10:06.256416 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.256397 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2wxrz" event={"ID":"4cfec505-8d15-44c1-af49-2328d48f3a70","Type":"ContainerStarted","Data":"66a6312e1a2bf69c2e8a43edb93dacb1852d9ff129852e52ef364e13a6cabdf5"} Jul 15 21:10:06.289787 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.289581 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k957b" podStartSLOduration=3.621560143 podStartE2EDuration="23.289569564s" totalImagesPullingTime="19.668009421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:06.261800801 +0000 UTC m=+24.388464780" watchObservedRunningTime="2026-07-15 21:10:06.289569564 +0000 UTC m=+24.416233543" Jul 15 21:10:06.313354 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.313298 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lqmdr" podStartSLOduration=3.361662101 podStartE2EDuration="23.313287393s" totalImagesPullingTime="19.951625292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:06.289868949 +0000 UTC m=+24.416532912" watchObservedRunningTime="2026-07-15 21:10:06.313287393 +0000 UTC m=+24.439951426" Jul 15 21:10:06.313835 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.313800 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2wxrz" podStartSLOduration=3.405929319 podStartE2EDuration="23.3137934s" totalImagesPullingTime="19.907864081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:06.313539488 +0000 UTC m=+24.440203467" watchObservedRunningTime="2026-07-15 21:10:06.3137934 +0000 UTC m=+24.440457379" Jul 15 21:10:06.410353 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:06.410263 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jdgz4" podStartSLOduration=3.528240041 podStartE2EDuration="23.410250045s" totalImagesPullingTime="19.882010004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:06.410110925 +0000 UTC m=+24.536774903" watchObservedRunningTime="2026-07-15 21:10:06.410250045 +0000 UTC m=+24.536914024" Jul 15 21:10:07.040631 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:07.040607 2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 21:10:07.259294 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:07.259242 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" event={"ID":"8ae616e3-61e9-41b2-b166-1790d9cba04d","Type":"ContainerStarted","Data":"48f55f4783348da01958bc23365ed9006934b2eebad6ff08053411242a9e2317"} Jul 15 21:10:07.260820 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:07.260787 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" event={"ID":"d00899e62aada42f2f0bdc458167583d","Type":"ContainerStarted","Data":"f4bea43011c6c97efc801701d5c56f455c13d413fb3ac460fc5c23fae3c36f3f"} Jul 15 21:10:07.277639 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:07.277601 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-220.ec2.internal" podStartSLOduration=23.277589801 podStartE2EDuration="23.277589801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:07.27708553 +0000 UTC m=+25.403749512" watchObservedRunningTime="2026-07-15 21:10:07.277589801 +0000 UTC m=+25.404253779" Jul 15 21:10:08.022924 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.022820 2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T21:10:07.04062902Z","UUID":"05cf5f72-2413-4c03-8bd0-4211a27a2819","Handler":null,"Name":"","Endpoint":""} Jul 15 21:10:08.024414 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.024384 2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 21:10:08.024514 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.024421 2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 21:10:08.138428 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.138401 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:08.138552 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:08.138530 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:08.138915 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.138892 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:08.139018 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:08.138993 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:08.139076 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.139063 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:08.139154 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:08.139129 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:08.265905 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.265824 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" event={"ID":"8ae616e3-61e9-41b2-b166-1790d9cba04d","Type":"ContainerStarted","Data":"c0e4358c643346d22ce98892e4851d6145c8db30c9b58411ea811663db27606c"} Jul 15 21:10:08.269313 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.269294 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:10:08.269775 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:08.269729 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"1c60315d445486d98753c2d5f52be23d1d63330f448c5d48e7ebfa486f906d08"} Jul 15 21:10:10.138446 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.138398 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:10.138791 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.138501 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:10.138791 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:10.138511 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:10.138791 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:10.138590 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:10.138791 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.138506 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:10.138791 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:10.138688 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:10.260413 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.260384 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:10:10.261099 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.261079 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:10:10.274046 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.274022 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:10:10.274402 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.274378 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2wxrz" Jul 15 21:10:10.276577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:10.276537 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dtkt" podStartSLOduration=5.411401635 podStartE2EDuration="27.276521952s" totalImagesPullingTime="21.865120317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:08.284271869 +0000 UTC m=+26.410935850" watchObservedRunningTime="2026-07-15 21:10:10.276521952 +0000 UTC m=+28.403185940" Jul 15 21:10:11.279614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:11.279450 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:10:11.280315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:11.280014 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"94f0fec553d33856db66cde07e729cfb0ad0be8731f835e186d7ecc10b0d1d79"} Jul 15 21:10:11.280372 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:11.280312 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:10:11.280419 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:11.280390 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:10:11.280498 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:11.280483 2553 scope.go:117] "RemoveContainer" containerID="fd5c6746861930e13ba27eafcaeb27c3f901e8f7f638fc11dbbe88e5e1930e40" Jul 15 21:10:11.282149 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:11.282117 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerStarted","Data":"d9d70cd4c9d0110952effa639eebcf4ba7196f052efab8e029d798029a1e82e3"} Jul 15 21:10:11.295054 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:11.295032 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:10:12.138017 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.137949 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:12.138017 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.137983 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:12.138297 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.138050 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:12.138297 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:12.138067 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:12.138297 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:12.138131 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:12.138297 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:12.138218 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:12.287062 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.287032 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca0d0d1d-4805-4ad6-9b4a-38b0e621affc" containerID="d9d70cd4c9d0110952effa639eebcf4ba7196f052efab8e029d798029a1e82e3" exitCode=0 Jul 15 21:10:12.287534 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.287111 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerDied","Data":"d9d70cd4c9d0110952effa639eebcf4ba7196f052efab8e029d798029a1e82e3"} Jul 15 21:10:12.291055 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.291040 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:10:12.291391 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.291368 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" event={"ID":"7d7cd9da-9939-42a9-9a3e-56129fc6f090","Type":"ContainerStarted","Data":"1798734e9cf1d230ef07fc4b2c2e78822c1ab4bed7a8952cf39a644bd3c45536"} Jul 15 21:10:12.291710 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.291697 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:10:12.305393 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.305376 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:10:12.349579 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:12.349533 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" podStartSLOduration=9.527352862 podStartE2EDuration="29.3495171s" totalImagesPullingTime="19.822164238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:12.34930146 +0000 UTC m=+30.475965484" watchObservedRunningTime="2026-07-15 21:10:12.3495171 +0000 UTC m=+30.476181082" Jul 15 21:10:13.156348 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:13.156315 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8cnsf"] Jul 15 21:10:13.156607 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:13.156442 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:13.156607 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:13.156557 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:13.156998 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:13.156971 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58zwt"] Jul 15 21:10:13.157136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:13.157074 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:13.157196 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:13.157164 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:13.157533 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:13.157503 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w5bkr"] Jul 15 21:10:13.157635 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:13.157588 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:13.157710 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:13.157689 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:14.296896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:14.296868 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca0d0d1d-4805-4ad6-9b4a-38b0e621affc" containerID="c6b1045fc603bd69bf0f32c8b4d85b9c0a7b8fcbb0841b3b6e1decf77c39afec" exitCode=0 Jul 15 21:10:14.297315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:14.296956 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerDied","Data":"c6b1045fc603bd69bf0f32c8b4d85b9c0a7b8fcbb0841b3b6e1decf77c39afec"} Jul 15 21:10:15.140916 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:15.140885 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:15.141054 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:15.140981 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:15.141054 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:15.140996 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:15.141054 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:15.141006 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:15.141152 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:15.141098 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:15.141191 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:15.141174 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:15.301245 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:15.301176 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca0d0d1d-4805-4ad6-9b4a-38b0e621affc" containerID="24810e51725da98594ca53ceee757d7f9c655af1f9da6f1432835b8af095be04" exitCode=0 Jul 15 21:10:15.301245 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:15.301216 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerDied","Data":"24810e51725da98594ca53ceee757d7f9c655af1f9da6f1432835b8af095be04"} Jul 15 21:10:16.082158 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:16.082123 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:16.082339 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:16.082191 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:16.082339 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.082277 2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:16.082424 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.082341 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs podName:bb98c88c-8490-4747-ac99-546a6159a8d3 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:48.082321702 +0000 UTC m=+66.208985665 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs") pod "network-metrics-daemon-8cnsf" (UID: "bb98c88c-8490-4747-ac99-546a6159a8d3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:10:16.082424 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.082280 2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:16.082424 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.082421 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret podName:47faff50-dced-41ee-b2da-a1ccb917bb9f nodeName:}" failed. No retries permitted until 2026-07-15 21:10:48.082405868 +0000 UTC m=+66.209069844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret") pod "global-pull-secret-syncer-w5bkr" (UID: "47faff50-dced-41ee-b2da-a1ccb917bb9f") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:10:16.182715 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:16.182679 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:16.182918 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.182874 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:10:16.182918 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.182895 2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:10:16.182918 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.182905 2553 projected.go:196] Error preparing data for projected volume kube-api-access-5hfqs for pod openshift-network-diagnostics/network-check-target-58zwt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:16.183102 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:16.182972 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs podName:ce4f6bd7-9e54-4730-9afc-5d1f14ed529e nodeName:}" failed. No retries permitted until 2026-07-15 21:10:48.182957029 +0000 UTC m=+66.309621003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5hfqs" (UniqueName: "kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs") pod "network-check-target-58zwt" (UID: "ce4f6bd7-9e54-4730-9afc-5d1f14ed529e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:10:16.995300 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:16.995214 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:17.086850 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:17.086809 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:17.141493 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:17.141471 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:17.141652 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:17.141559 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:17.141652 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:17.141591 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:17.141652 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:17.141599 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5bkr" podUID="47faff50-dced-41ee-b2da-a1ccb917bb9f" Jul 15 21:10:17.141819 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:17.141681 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58zwt" podUID="ce4f6bd7-9e54-4730-9afc-5d1f14ed529e" Jul 15 21:10:17.141819 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:17.141785 2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8cnsf" podUID="bb98c88c-8490-4747-ac99-546a6159a8d3" Jul 15 21:10:17.481487 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:17.481450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:17.581639 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:17.581610 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:17.931455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:17.931418 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:18.031444 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.031364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:18.132331 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.132300 2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-220.ec2.internal" event="NodeReady" Jul 15 21:10:18.132486 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.132447 2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 21:10:18.192323 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.192290 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jmv6d"] Jul 15 21:10:18.194806 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.194783 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.200871 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.200681 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 21:10:18.201016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.200879 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 21:10:18.201016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.200932 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xcfql\"" Jul 15 21:10:18.203114 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.202702 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m24rn"] Jul 15 21:10:18.204585 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.204566 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.207110 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.207091 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 21:10:18.207203 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.207136 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 21:10:18.207265 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.207254 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9kxvh\"" Jul 15 21:10:18.207314 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.207252 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 21:10:18.213797 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.213729 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jmv6d"] Jul 15 21:10:18.220193 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.220169 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m24rn"] Jul 15 21:10:18.299071 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.298985 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.299071 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.299050 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6c8\" (UniqueName: \"kubernetes.io/projected/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-kube-api-access-ck6c8\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.299262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.299109 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af8aa079-6a91-4be0-ad4a-65c8de7ccf07-cert\") pod \"ingress-canary-m24rn\" (UID: \"af8aa079-6a91-4be0-ad4a-65c8de7ccf07\") " pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.299262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.299188 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-config-volume\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.299262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.299212 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-tmp-dir\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.299262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.299235 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shtq\" (UniqueName: \"kubernetes.io/projected/af8aa079-6a91-4be0-ad4a-65c8de7ccf07-kube-api-access-9shtq\") pod \"ingress-canary-m24rn\" (UID: \"af8aa079-6a91-4be0-ad4a-65c8de7ccf07\") " pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.365351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.365311 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:18.400179 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400141 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.400179 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400202 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6c8\" (UniqueName: \"kubernetes.io/projected/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-kube-api-access-ck6c8\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.400433 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400308 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af8aa079-6a91-4be0-ad4a-65c8de7ccf07-cert\") pod \"ingress-canary-m24rn\" (UID: \"af8aa079-6a91-4be0-ad4a-65c8de7ccf07\") " pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.400433 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:18.400319 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 21:10:18.400433 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400366 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-config-volume\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.400433 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:18.400401 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls podName:2cdb38bf-4fe4-4d06-8e86-109d1e447b69 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:18.900378015 +0000 UTC m=+37.027041978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls") pod "dns-default-jmv6d" (UID: "2cdb38bf-4fe4-4d06-8e86-109d1e447b69") : secret "dns-default-metrics-tls" not found Jul 15 21:10:18.400597 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400445 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-tmp-dir\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.400597 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400483 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shtq\" (UniqueName: \"kubernetes.io/projected/af8aa079-6a91-4be0-ad4a-65c8de7ccf07-kube-api-access-9shtq\") pod \"ingress-canary-m24rn\" (UID: \"af8aa079-6a91-4be0-ad4a-65c8de7ccf07\") " pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.400982 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400959 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-config-volume\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.401065 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.400984 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-tmp-dir\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.405374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.405231 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/af8aa079-6a91-4be0-ad4a-65c8de7ccf07-cert\") pod \"ingress-canary-m24rn\" (UID: \"af8aa079-6a91-4be0-ad4a-65c8de7ccf07\") " pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.410209 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.410183 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6c8\" (UniqueName: \"kubernetes.io/projected/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-kube-api-access-ck6c8\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.410441 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.410419 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shtq\" (UniqueName: \"kubernetes.io/projected/af8aa079-6a91-4be0-ad4a-65c8de7ccf07-kube-api-access-9shtq\") pod \"ingress-canary-m24rn\" (UID: \"af8aa079-6a91-4be0-ad4a-65c8de7ccf07\") " pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.430486 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.430455 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:18.514434 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.514402 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m24rn" Jul 15 21:10:18.904157 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:18.904119 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:18.904345 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:18.904268 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 21:10:18.904345 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:18.904333 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls podName:2cdb38bf-4fe4-4d06-8e86-109d1e447b69 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:19.90431618 +0000 UTC m=+38.030980159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls") pod "dns-default-jmv6d" (UID: "2cdb38bf-4fe4-4d06-8e86-109d1e447b69") : secret "dns-default-metrics-tls" not found Jul 15 21:10:19.138857 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.138190 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:19.138857 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.138385 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:19.138857 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.138494 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:19.147883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.144938 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s56kw\"" Jul 15 21:10:19.147883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.145391 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 21:10:19.147883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.146031 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 21:10:19.147883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.146876 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2pnr\"" Jul 15 21:10:19.147883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.147004 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 21:10:19.147883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.147127 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 21:10:19.209155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.209072 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m24rn"] Jul 15 21:10:19.212472 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:19.212444 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf8aa079_6a91_4be0_ad4a_65c8de7ccf07.slice/crio-a059796c0f6a978f34cc1640b121eef3ac84477adfef153b7bde04a21568d4c1 WatchSource:0}: Error finding container a059796c0f6a978f34cc1640b121eef3ac84477adfef153b7bde04a21568d4c1: Status 404 returned error can't find the container with id a059796c0f6a978f34cc1640b121eef3ac84477adfef153b7bde04a21568d4c1 Jul 15 21:10:19.315600 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.315571 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerStarted","Data":"92e930e6a2ae9db6a169ee21974733d07cf24e49fb3fadc394cfedab4ec1a078"} Jul 15 21:10:19.316692 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.316663 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m24rn" event={"ID":"af8aa079-6a91-4be0-ad4a-65c8de7ccf07","Type":"ContainerStarted","Data":"a059796c0f6a978f34cc1640b121eef3ac84477adfef153b7bde04a21568d4c1"} Jul 15 21:10:19.912505 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:19.912474 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:19.912665 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:19.912649 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 21:10:19.912726 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:19.912715 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls podName:2cdb38bf-4fe4-4d06-8e86-109d1e447b69 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:21.912696996 +0000 UTC m=+40.039360954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls") pod "dns-default-jmv6d" (UID: "2cdb38bf-4fe4-4d06-8e86-109d1e447b69") : secret "dns-default-metrics-tls" not found Jul 15 21:10:20.322583 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:20.322551 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca0d0d1d-4805-4ad6-9b4a-38b0e621affc" containerID="92e930e6a2ae9db6a169ee21974733d07cf24e49fb3fadc394cfedab4ec1a078" exitCode=0 Jul 15 21:10:20.322583 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:20.322580 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca0d0d1d-4805-4ad6-9b4a-38b0e621affc" containerID="84e83c604c5475e21610e4868878f7314286c66fe7c53a01abfe546be74843dc" exitCode=0 Jul 15 21:10:20.323184 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:20.322623 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerDied","Data":"92e930e6a2ae9db6a169ee21974733d07cf24e49fb3fadc394cfedab4ec1a078"} Jul 15 21:10:20.323184 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:20.322680 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerDied","Data":"84e83c604c5475e21610e4868878f7314286c66fe7c53a01abfe546be74843dc"} Jul 15 21:10:21.243832 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:21.243600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:21.331925 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:21.331894 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vchqc" event={"ID":"ca0d0d1d-4805-4ad6-9b4a-38b0e621affc","Type":"ContainerStarted","Data":"b40b020694014a950f44d111d9bf21c7e9255bf68b637991631984622bf2d907"} Jul 15 21:10:21.343874 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:21.343845 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:21.355826 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:21.355784 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vchqc" podStartSLOduration=8.132593705 podStartE2EDuration="38.3557533s" totalImagesPullingTime="30.223159595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:21.354201044 +0000 UTC m=+39.480865022" watchObservedRunningTime="2026-07-15 21:10:21.3557533 +0000 UTC m=+39.482417280" Jul 15 21:10:21.545447 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:21.545357 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:21.648215 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:21.648178 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:21.936970 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:21.936317 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:21.936970 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:21.936517 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 21:10:21.936970 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:21.936587 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls podName:2cdb38bf-4fe4-4d06-8e86-109d1e447b69 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:25.936568151 +0000 UTC m=+44.063232130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls") pod "dns-default-jmv6d" (UID: "2cdb38bf-4fe4-4d06-8e86-109d1e447b69") : secret "dns-default-metrics-tls" not found Jul 15 21:10:22.335550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.335516 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m24rn" event={"ID":"af8aa079-6a91-4be0-ad4a-65c8de7ccf07","Type":"ContainerStarted","Data":"79a042cf27cc8502b8388e0e166296fdae483024394587d69f2400434ddb810f"} Jul 15 21:10:22.352872 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.352832 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m24rn" podStartSLOduration=1.619475101 podStartE2EDuration="4.352819435s" totalImagesPullingTime="2.733344334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:22.351515833 +0000 UTC m=+40.478179815" watchObservedRunningTime="2026-07-15 21:10:22.352819435 +0000 UTC m=+40.479483414" Jul 15 21:10:22.862045 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.861992 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:22.953403 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.953368 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:22.964184 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.964156 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bjbs6"] Jul 15 21:10:22.970280 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.970264 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:22.973968 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.973948 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 21:10:22.974073 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.973988 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nj8l9\"" Jul 15 21:10:22.974420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.974402 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 21:10:22.974539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.974471 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 21:10:22.974539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:22.974498 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 21:10:23.041932 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.041907 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjbs6"] Jul 15 21:10:23.088831 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.088799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:23.143614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.143549 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/013328ec-a9d4-457e-b5d2-6944a2196aa5-crio-socket\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.143614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.143580 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.143614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.143608 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/013328ec-a9d4-457e-b5d2-6944a2196aa5-data-volume\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.143800 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.143634 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.143800 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.143681 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjk9\" (UniqueName: \"kubernetes.io/projected/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-api-access-qqjk9\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.189472 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.189448 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:23.244181 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.244160 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/013328ec-a9d4-457e-b5d2-6944a2196aa5-crio-socket\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.244272 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.244189 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.244272 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.244221 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/013328ec-a9d4-457e-b5d2-6944a2196aa5-data-volume\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.244272 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.244238 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.244272 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.244258 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjk9\" (UniqueName: \"kubernetes.io/projected/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-api-access-qqjk9\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.244435 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.244326 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 21:10:23.244435 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.244333 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 21:10:23.244435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.244389 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/013328ec-a9d4-457e-b5d2-6944a2196aa5-crio-socket\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.244435 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.244405 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:23.744388095 +0000 UTC m=+41.871052073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : configmap "kube-rbac-proxy" not found Jul 15 21:10:23.244596 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.244443 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:23.744424209 +0000 UTC m=+41.871088182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : secret "insights-runtime-extractor-tls" not found Jul 15 21:10:23.244596 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.244539 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/013328ec-a9d4-457e-b5d2-6944a2196aa5-data-volume\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.261738 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.261718 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjk9\" (UniqueName: \"kubernetes.io/projected/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-api-access-qqjk9\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.326141 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.326106 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:23.329773 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.329743 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg"] Jul 15 21:10:23.350548 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.350527 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg"] Jul 15 21:10:23.350852 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.350602 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" Jul 15 21:10:23.353055 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.353027 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vjtd2\"" Jul 15 21:10:23.353169 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.353150 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 15 21:10:23.353928 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.353915 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 15 21:10:23.406276 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.406211 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:23.446194 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.446169 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tjnx\" (UniqueName: \"kubernetes.io/projected/afc5e170-3cf3-4510-9498-101aad6b7c49-kube-api-access-2tjnx\") pod \"migrator-7f59b54f6c-z6khg\" (UID: \"afc5e170-3cf3-4510-9498-101aad6b7c49\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" Jul 15 21:10:23.516428 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.516397 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:23.547467 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.547437 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tjnx\" (UniqueName: \"kubernetes.io/projected/afc5e170-3cf3-4510-9498-101aad6b7c49-kube-api-access-2tjnx\") pod \"migrator-7f59b54f6c-z6khg\" (UID: \"afc5e170-3cf3-4510-9498-101aad6b7c49\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" Jul 15 21:10:23.561946 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.561919 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tjnx\" (UniqueName: \"kubernetes.io/projected/afc5e170-3cf3-4510-9498-101aad6b7c49-kube-api-access-2tjnx\") pod \"migrator-7f59b54f6c-z6khg\" (UID: \"afc5e170-3cf3-4510-9498-101aad6b7c49\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" Jul 15 21:10:23.650179 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.650148 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:23.658738 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.658684 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" Jul 15 21:10:23.749724 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.749696 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.749873 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.749748 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:23.749873 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.749864 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 21:10:23.749977 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.749919 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:24.749904993 +0000 UTC m=+42.876568951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : configmap "kube-rbac-proxy" not found Jul 15 21:10:23.749977 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.749864 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 21:10:23.750088 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:23.749990 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:24.749977796 +0000 UTC m=+42.876641772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : secret "insights-runtime-extractor-tls" not found Jul 15 21:10:23.781550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:23.781522 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg"] Jul 15 21:10:24.341070 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:24.341032 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" event={"ID":"afc5e170-3cf3-4510-9498-101aad6b7c49","Type":"ContainerStarted","Data":"ed1d7ee01ca780f52cfff9a32f21198f1df3473b45fd6d03f793dcec36ef14a9"} Jul 15 21:10:24.757110 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:24.757034 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:24.757110 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:24.757080 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:24.757462 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:24.757172 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 21:10:24.757462 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:24.757190 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 21:10:24.757462 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:24.757220 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:26.757206773 +0000 UTC m=+44.883870734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : configmap "kube-rbac-proxy" not found Jul 15 21:10:24.757462 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:24.757244 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:26.757231711 +0000 UTC m=+44.883895670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : secret "insights-runtime-extractor-tls" not found Jul 15 21:10:24.949271 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:24.949243 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lqmdr_00459e80-47cb-4df5-a7d0-4a6c6d0a173c/dns-node-resolver/0.log" Jul 15 21:10:25.154420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.154391 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-98zrg"] Jul 15 21:10:25.173386 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.173357 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-98zrg"] Jul 15 21:10:25.173563 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.173466 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.176199 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.176175 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 21:10:25.176199 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.176194 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 21:10:25.176986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.176966 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 21:10:25.177078 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.177000 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 21:10:25.177078 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.177011 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-dwp6r\"" Jul 15 21:10:25.264166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.264130 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:25.361646 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.361616 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/217576e2-6117-438a-8f01-dc121d159a0f-signing-cabundle\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.361823 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.361683 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ns2\" (UniqueName: \"kubernetes.io/projected/217576e2-6117-438a-8f01-dc121d159a0f-kube-api-access-d9ns2\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.361823 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.361727 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/217576e2-6117-438a-8f01-dc121d159a0f-signing-key\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.367509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.367487 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:25.463056 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.462982 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/217576e2-6117-438a-8f01-dc121d159a0f-signing-cabundle\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.463056 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.463051 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ns2\" (UniqueName: \"kubernetes.io/projected/217576e2-6117-438a-8f01-dc121d159a0f-kube-api-access-d9ns2\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.463306 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.463125 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/217576e2-6117-438a-8f01-dc121d159a0f-signing-key\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.463834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.463811 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/217576e2-6117-438a-8f01-dc121d159a0f-signing-cabundle\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.465812 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.465792 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/217576e2-6117-438a-8f01-dc121d159a0f-signing-key\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.474141 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.474122 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ns2\" (UniqueName: \"kubernetes.io/projected/217576e2-6117-438a-8f01-dc121d159a0f-kube-api-access-d9ns2\") pod \"service-ca-79f68b9f4f-98zrg\" (UID: \"217576e2-6117-438a-8f01-dc121d159a0f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.482874 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.482855 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" Jul 15 21:10:25.616639 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.616611 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-98zrg"] Jul 15 21:10:25.751681 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.751611 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k957b_82ffd961-41ee-44d3-b38f-c74dafaa59ca/node-ca/0.log" Jul 15 21:10:25.888583 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:25.888553 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217576e2_6117_438a_8f01_dc121d159a0f.slice/crio-e76662b1e793fd6a9b4bc5f3116d0ab6f55274c2c78b1c66b5e6c4218169558f WatchSource:0}: Error finding container e76662b1e793fd6a9b4bc5f3116d0ab6f55274c2c78b1c66b5e6c4218169558f: Status 404 returned error can't find the container with id e76662b1e793fd6a9b4bc5f3116d0ab6f55274c2c78b1c66b5e6c4218169558f Jul 15 21:10:25.967031 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:25.967008 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:25.967170 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:25.967152 2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 21:10:25.967217 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:25.967210 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls podName:2cdb38bf-4fe4-4d06-8e86-109d1e447b69 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:33.967195605 +0000 UTC m=+52.093859563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls") pod "dns-default-jmv6d" (UID: "2cdb38bf-4fe4-4d06-8e86-109d1e447b69") : secret "dns-default-metrics-tls" not found Jul 15 21:10:26.197207 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:26.197174 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:26.347779 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:26.347739 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" event={"ID":"afc5e170-3cf3-4510-9498-101aad6b7c49","Type":"ContainerStarted","Data":"ef04e3bf572c13878df7a887240ef8a1c76548f3c04ac4c2c4c8fe39de42d928"} Jul 15 21:10:26.348798 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:26.348776 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" event={"ID":"217576e2-6117-438a-8f01-dc121d159a0f","Type":"ContainerStarted","Data":"e76662b1e793fd6a9b4bc5f3116d0ab6f55274c2c78b1c66b5e6c4218169558f"} Jul 15 21:10:26.749154 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:26.749080 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m24rn_af8aa079-6a91-4be0-ad4a-65c8de7ccf07/serve-healthcheck-canary/0.log" Jul 15 21:10:26.772200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:26.772168 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:26.772342 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:26.772220 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:26.772342 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:26.772322 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 21:10:26.772342 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:26.772332 2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 21:10:26.772426 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:26.772382 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:30.772368157 +0000 UTC m=+48.899032137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : configmap "kube-rbac-proxy" not found Jul 15 21:10:26.772426 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:26.772397 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:30.772391296 +0000 UTC m=+48.899055254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : secret "insights-runtime-extractor-tls" not found Jul 15 21:10:27.353125 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:27.353094 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" event={"ID":"afc5e170-3cf3-4510-9498-101aad6b7c49","Type":"ContainerStarted","Data":"bfe0e9aa6e9fe41f76b70ca437b299e75a4af20d068b22bb0400e30fc9b37a4d"} Jul 15 21:10:27.370887 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:27.370838 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-z6khg" podStartSLOduration=1.919417439 podStartE2EDuration="4.370823784s" totalImagesPullingTime="2.451406345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:27.370322554 +0000 UTC m=+45.496986533" watchObservedRunningTime="2026-07-15 21:10:27.370823784 +0000 UTC m=+45.497487764" Jul 15 21:10:29.361377 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:29.361344 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" event={"ID":"217576e2-6117-438a-8f01-dc121d159a0f","Type":"ContainerStarted","Data":"c5bd95599332d0ef90de125ee5e50ff8e1eddd16a6590b0504782c35d9bfd6d2"} Jul 15 21:10:29.362689 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:29.362656 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:29.379115 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:29.379067 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-98zrg" podStartSLOduration=1.8462766720000001 podStartE2EDuration="4.379049525s" totalImagesPullingTime="2.532772853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:29.379048563 +0000 UTC m=+47.505712557" watchObservedRunningTime="2026-07-15 21:10:29.379049525 +0000 UTC m=+47.505713501" Jul 15 21:10:29.466619 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:29.466593 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:29.793516 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:29.793427 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:29.950294 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:29.950259 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.080835 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:30.080802 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.502909 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:30.502828 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:30.806905 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:30.806876 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:30.807111 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:30.807089 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:30.807247 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:30.807226 2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 21:10:30.807316 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:10:30.807304 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls podName:013328ec-a9d4-457e-b5d2-6944a2196aa5 nodeName:}" failed. No retries permitted until 2026-07-15 21:10:38.807286832 +0000 UTC m=+56.933950808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bjbs6" (UID: "013328ec-a9d4-457e-b5d2-6944a2196aa5") : secret "insights-runtime-extractor-tls" not found Jul 15 21:10:30.807456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:30.807435 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013328ec-a9d4-457e-b5d2-6944a2196aa5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:30.878519 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:30.878486 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:31.459173 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:31.459137 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:31.662918 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:31.662876 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:32.050107 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:32.050073 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:32.263514 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:32.263471 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:32.680630 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:32.680596 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:33.054443 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:33.054413 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:33.449245 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:33.449166 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:33.831393 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:33.831347 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:33.894530 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:33.894488 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:34.028964 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.028936 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:34.031157 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.031134 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cdb38bf-4fe4-4d06-8e86-109d1e447b69-metrics-tls\") pod \"dns-default-jmv6d\" (UID: \"2cdb38bf-4fe4-4d06-8e86-109d1e447b69\") " pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:34.052721 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.052693 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:34.105629 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.105556 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:34.235580 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.235549 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jmv6d"] Jul 15 21:10:34.238677 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:34.238650 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdb38bf_4fe4_4d06_8e86_109d1e447b69.slice/crio-b5248305ac6c0e57f79cf632d4168d651e79e29e977f8000b69e77523c4c8674 WatchSource:0}: Error finding container b5248305ac6c0e57f79cf632d4168d651e79e29e977f8000b69e77523c4c8674: Status 404 returned error can't find the container with id b5248305ac6c0e57f79cf632d4168d651e79e29e977f8000b69e77523c4c8674 Jul 15 21:10:34.373992 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.373914 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmv6d" event={"ID":"2cdb38bf-4fe4-4d06-8e86-109d1e447b69","Type":"ContainerStarted","Data":"b5248305ac6c0e57f79cf632d4168d651e79e29e977f8000b69e77523c4c8674"} Jul 15 21:10:34.448162 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.448128 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:34.518191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.518156 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:34.845639 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:34.845600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:35.480344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:35.480306 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:35.582005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:35.581960 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:35.851303 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:35.851269 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.181425 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:36.181358 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.282475 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:36.282450 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.386102 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:36.386070 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:36.490614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:36.490145 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:37.060210 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.060172 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:37.158169 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.158141 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:37.257742 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.257710 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:37.384537 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.384467 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmv6d" event={"ID":"2cdb38bf-4fe4-4d06-8e86-109d1e447b69","Type":"ContainerStarted","Data":"f34d502e58c6ed8c89a1ebb64feba68e0f218a32ca532cc4df8c8eef39427947"} Jul 15 21:10:37.384537 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.384513 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jmv6d" event={"ID":"2cdb38bf-4fe4-4d06-8e86-109d1e447b69","Type":"ContainerStarted","Data":"31adf230790a4c9812c528e0aa53cbac703822d0007a05a2db5e22edc28d62e5"} Jul 15 21:10:37.384725 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.384590 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:37.405003 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.404965 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jmv6d" podStartSLOduration=17.311549796 podStartE2EDuration="19.404951156s" totalImagesPullingTime="2.09340136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:37.403665323 +0000 UTC m=+55.530329302" watchObservedRunningTime="2026-07-15 21:10:37.404951156 +0000 UTC m=+55.531615135" Jul 15 21:10:37.477199 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:37.477170 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:38.059511 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:38.059476 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:38.654370 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:38.654343 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:38.851254 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:38.851218 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:38.868006 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:38.867982 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:38.870157 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:38.870134 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013328ec-a9d4-457e-b5d2-6944a2196aa5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bjbs6\" (UID: \"013328ec-a9d4-457e-b5d2-6944a2196aa5\") " pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:38.880128 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:38.880110 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bjbs6" Jul 15 21:10:39.009711 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:39.009684 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bjbs6"] Jul 15 21:10:39.012931 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:39.012898 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013328ec_a9d4_457e_b5d2_6944a2196aa5.slice/crio-30b09399ea054b405eab5278dccd20c37f39961571efc4415e8ae6f07ce6fdac WatchSource:0}: Error finding container 30b09399ea054b405eab5278dccd20c37f39961571efc4415e8ae6f07ce6fdac: Status 404 returned error can't find the container with id 30b09399ea054b405eab5278dccd20c37f39961571efc4415e8ae6f07ce6fdac Jul 15 21:10:39.062743 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:39.062715 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:39.392595 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:39.392569 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjbs6" event={"ID":"013328ec-a9d4-457e-b5d2-6944a2196aa5","Type":"ContainerStarted","Data":"30b09399ea054b405eab5278dccd20c37f39961571efc4415e8ae6f07ce6fdac"} Jul 15 21:10:39.812193 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:39.812158 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:39.924561 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:39.924527 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.065000 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:40.064921 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.274785 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:40.274706 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.373927 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:40.373822 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.481218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:40.481183 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.829603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:40.829565 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:40.903776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:40.903731 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:41.237454 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:41.237377 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:41.643521 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:41.643490 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:41.744234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:41.744196 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.475902 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:42.475858 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:42.569951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:42.569913 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:44.316739 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:44.316715 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zqbh7" Jul 15 21:10:44.410655 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:44.410630 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjbs6" event={"ID":"013328ec-a9d4-457e-b5d2-6944a2196aa5","Type":"ContainerStarted","Data":"2ebb7d5d473e87adc82a060e1da64e10dfb458e092354bddc410d61cb8eda4f4"} Jul 15 21:10:46.072696 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.072664 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z"] Jul 15 21:10:46.081666 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.081643 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.086489 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.086468 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 15 21:10:46.086896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.086876 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 15 21:10:46.087018 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.086932 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 15 21:10:46.087930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.087910 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 21:10:46.088043 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.087916 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 15 21:10:46.089877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.089307 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 21:10:46.089877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.089314 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 21:10:46.115664 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.115631 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z"] Jul 15 21:10:46.118987 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.118962 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-hub\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.119098 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.118993 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.119098 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.119085 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.119192 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.119119 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgzk7\" (UniqueName: \"kubernetes.io/projected/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-kube-api-access-pgzk7\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.119246 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.119210 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-ca\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.119246 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.119237 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.219724 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.219688 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-ca\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.219944 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.219735 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.220021 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.219954 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-hub\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.220021 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.219994 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.220118 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.220053 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.220118 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.220081 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgzk7\" (UniqueName: \"kubernetes.io/projected/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-kube-api-access-pgzk7\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.220976 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.220926 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.222603 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.222572 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-ca\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.222918 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.222895 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.223005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.222922 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.223005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.222980 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-hub\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.244300 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.244272 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgzk7\" (UniqueName: \"kubernetes.io/projected/cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b-kube-api-access-pgzk7\") pod \"cluster-proxy-proxy-agent-76f8d776cc-sxb5z\" (UID: \"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.248085 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.248055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:46.393625 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.393546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:46.399348 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.399324 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" Jul 15 21:10:46.492663 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.492625 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:46.594349 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:46.594300 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:47.390837 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:47.390789 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jmv6d" Jul 15 21:10:47.452261 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:47.452236 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z"] Jul 15 21:10:47.455157 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:47.455134 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcefa3a3f_c4c2_4102_b54a_3db7ed7ae15b.slice/crio-7a92205fc60e12c34b46542f702b2e641eec58bb215d793267d2315db34786ba WatchSource:0}: Error finding container 7a92205fc60e12c34b46542f702b2e641eec58bb215d793267d2315db34786ba: Status 404 returned error can't find the container with id 7a92205fc60e12c34b46542f702b2e641eec58bb215d793267d2315db34786ba Jul 15 21:10:48.132786 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.132736 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:48.132948 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.132826 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:48.135351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.135311 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 21:10:48.136040 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.136017 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 21:10:48.146825 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.146801 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb98c88c-8490-4747-ac99-546a6159a8d3-metrics-certs\") pod \"network-metrics-daemon-8cnsf\" (UID: \"bb98c88c-8490-4747-ac99-546a6159a8d3\") " pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:48.147714 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.147671 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/47faff50-dced-41ee-b2da-a1ccb917bb9f-original-pull-secret\") pod \"global-pull-secret-syncer-w5bkr\" (UID: \"47faff50-dced-41ee-b2da-a1ccb917bb9f\") " pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:48.233935 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.233478 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:48.240669 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.240452 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 21:10:48.247148 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.246899 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 21:10:48.257043 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.256989 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfqs\" (UniqueName: \"kubernetes.io/projected/ce4f6bd7-9e54-4730-9afc-5d1f14ed529e-kube-api-access-5hfqs\") pod \"network-check-target-58zwt\" (UID: \"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e\") " pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:48.263326 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.263180 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f2pnr\"" Jul 15 21:10:48.270565 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.270541 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s56kw\"" Jul 15 21:10:48.270664 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.270657 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8cnsf" Jul 15 21:10:48.279277 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.279211 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:48.289528 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.289362 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5bkr" Jul 15 21:10:48.433702 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.433614 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjbs6" event={"ID":"013328ec-a9d4-457e-b5d2-6944a2196aa5","Type":"ContainerStarted","Data":"a0714797070aeb223ee6394ebb9ec2e76e8f933c2ee9f50b85b28af4518880f6"} Jul 15 21:10:48.436452 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.436396 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" event={"ID":"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b","Type":"ContainerStarted","Data":"7a92205fc60e12c34b46542f702b2e641eec58bb215d793267d2315db34786ba"} Jul 15 21:10:48.448219 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.447190 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8cnsf"] Jul 15 21:10:48.453615 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:48.453569 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb98c88c_8490_4747_ac99_546a6159a8d3.slice/crio-234444c5812a8e4673c04b0adbb02368d3181fcbe68923a09d602cbfe42644a8 WatchSource:0}: Error finding container 234444c5812a8e4673c04b0adbb02368d3181fcbe68923a09d602cbfe42644a8: Status 404 returned error can't find the container with id 234444c5812a8e4673c04b0adbb02368d3181fcbe68923a09d602cbfe42644a8 Jul 15 21:10:48.464790 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.464747 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58zwt"] Jul 15 21:10:48.470864 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:48.470832 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4f6bd7_9e54_4730_9afc_5d1f14ed529e.slice/crio-b78d587f028526d603c8e547efce1c237a047454f8a449ce64cae44b37b0e4b0 WatchSource:0}: Error finding container b78d587f028526d603c8e547efce1c237a047454f8a449ce64cae44b37b0e4b0: Status 404 returned error can't find the container with id b78d587f028526d603c8e547efce1c237a047454f8a449ce64cae44b37b0e4b0 Jul 15 21:10:48.483205 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:48.483149 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w5bkr"] Jul 15 21:10:48.486200 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:10:48.486175 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47faff50_dced_41ee_b2da_a1ccb917bb9f.slice/crio-f3b1bb062d205542d2915c1659595fc47b1b6c5325da1a0ee1275d14acb75738 WatchSource:0}: Error finding container f3b1bb062d205542d2915c1659595fc47b1b6c5325da1a0ee1275d14acb75738: Status 404 returned error can't find the container with id f3b1bb062d205542d2915c1659595fc47b1b6c5325da1a0ee1275d14acb75738 Jul 15 21:10:49.446751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:49.446700 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w5bkr" event={"ID":"47faff50-dced-41ee-b2da-a1ccb917bb9f","Type":"ContainerStarted","Data":"f3b1bb062d205542d2915c1659595fc47b1b6c5325da1a0ee1275d14acb75738"} Jul 15 21:10:49.448730 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:49.448681 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58zwt" event={"ID":"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e","Type":"ContainerStarted","Data":"b78d587f028526d603c8e547efce1c237a047454f8a449ce64cae44b37b0e4b0"} Jul 15 21:10:49.453973 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:49.453940 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8cnsf" event={"ID":"bb98c88c-8490-4747-ac99-546a6159a8d3","Type":"ContainerStarted","Data":"234444c5812a8e4673c04b0adbb02368d3181fcbe68923a09d602cbfe42644a8"} Jul 15 21:10:50.057068 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:50.057014 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:50.159122 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:50.159050 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:51.252582 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:51.252546 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:51.316551 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:51.316495 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:52.322791 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:52.322747 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:52.423960 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:52.423924 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:52.500973 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:52.500937 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:52.567283 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:52.567248 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:53.231428 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:53.231395 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:53.331520 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:53.331470 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:53.393597 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:53.393558 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:54.649387 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:54.649352 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:54.749043 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:54.749013 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:55.393332 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:55.393294 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:55.496463 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:55.496428 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:55.762218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:55.762122 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:56.480674 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.480588 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w5bkr" event={"ID":"47faff50-dced-41ee-b2da-a1ccb917bb9f","Type":"ContainerStarted","Data":"ee4f0be80375e8c0ea0a66a4f51af27091937c1546316b2b7beb0665e8f66f31"} Jul 15 21:10:56.482194 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.482166 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58zwt" event={"ID":"ce4f6bd7-9e54-4730-9afc-5d1f14ed529e","Type":"ContainerStarted","Data":"eb1b2900386e18ac2f5c480de1724e100746b08fa91891e64d49e7d0878ded2a"} Jul 15 21:10:56.482319 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.482242 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:10:56.483869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.483835 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8cnsf" event={"ID":"bb98c88c-8490-4747-ac99-546a6159a8d3","Type":"ContainerStarted","Data":"3e400b988594692b9b960fc0ba6f0e8ad5d7c843b4e33041b5d9ff6dc23bdaf1"} Jul 15 21:10:56.483989 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.483882 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8cnsf" event={"ID":"bb98c88c-8490-4747-ac99-546a6159a8d3","Type":"ContainerStarted","Data":"899847d87058420af95892a826ff08b7577b55a30bd9f1cdf434dbcd35e7f306"} Jul 15 21:10:56.485673 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.485654 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bjbs6" event={"ID":"013328ec-a9d4-457e-b5d2-6944a2196aa5","Type":"ContainerStarted","Data":"b7727d7ab99eb14d70b08300f559c72d9e3564a94f4c528e77c5214078abd36e"} Jul 15 21:10:56.487400 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.487383 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" event={"ID":"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b","Type":"ContainerStarted","Data":"f79fd8d9d2518621a1938852bf9877fe4b9425ea53c5ae3a412f50838157ad84"} Jul 15 21:10:56.487480 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.487408 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" event={"ID":"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b","Type":"ContainerStarted","Data":"7076ad85f88ce2144b38d2c62caeeb9f733c37ccb3d0044f30ec06330c07604f"} Jul 15 21:10:56.487480 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.487420 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" event={"ID":"cefa3a3f-c4c2-4102-b54a-3db7ed7ae15b","Type":"ContainerStarted","Data":"6f1320e08f6a75de9e84a865a26fc263f2660991f7e4f35c640003033efcd276"} Jul 15 21:10:56.497566 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.497530 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w5bkr" podStartSLOduration=65.854932284 podStartE2EDuration="1m13.497519892s" totalImagesPullingTime="7.642587608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:56.496849038 +0000 UTC m=+74.623513018" watchObservedRunningTime="2026-07-15 21:10:56.497519892 +0000 UTC m=+74.624183870" Jul 15 21:10:56.516071 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.516031 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8cnsf" podStartSLOduration=65.863107355 podStartE2EDuration="1m13.516016786s" totalImagesPullingTime="7.652909431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:56.512550659 +0000 UTC m=+74.639214650" watchObservedRunningTime="2026-07-15 21:10:56.516016786 +0000 UTC m=+74.642680744" Jul 15 21:10:56.530656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.530614 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-58zwt" podStartSLOduration=65.878515149 podStartE2EDuration="1m13.530599211s" totalImagesPullingTime="7.652084062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:09:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:56.530477439 +0000 UTC m=+74.657141418" watchObservedRunningTime="2026-07-15 21:10:56.530599211 +0000 UTC m=+74.657263193" Jul 15 21:10:56.539273 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.539236 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:56.552951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.552918 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bjbs6" podStartSLOduration=17.626425456 podStartE2EDuration="34.552904799s" totalImagesPullingTime="16.926479343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:10:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:56.552608625 +0000 UTC m=+74.679272605" watchObservedRunningTime="2026-07-15 21:10:56.552904799 +0000 UTC m=+74.679568796" Jul 15 21:10:56.576645 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.576614 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f8d776cc-sxb5z" podStartSLOduration=1.9032524 podStartE2EDuration="10.576603343s" totalImagesPullingTime="8.673350943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:10:56.576559095 +0000 UTC m=+74.703223074" watchObservedRunningTime="2026-07-15 21:10:56.576603343 +0000 UTC m=+74.703267322" Jul 15 21:10:56.622960 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.622938 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:56.717724 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.717694 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:56.821398 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:56.821365 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.778189 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:58.778156 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:10:58.882028 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:10:58.881992 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:00.286426 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.286394 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6466fbd457-s4hgl"] Jul 15 21:11:00.289247 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.289223 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.296613 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.296573 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 21:11:00.296789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.296748 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 21:11:00.296980 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.296964 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 21:11:00.297077 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.297012 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 21:11:00.297970 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.297948 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 21:11:00.298104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.298088 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 21:11:00.298667 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.298654 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 21:11:00.298667 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.298665 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-64fj2\"" Jul 15 21:11:00.310923 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.310897 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 21:11:00.325207 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.325175 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6466fbd457-s4hgl"] Jul 15 21:11:00.420943 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.420914 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-trusted-ca-bundle\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.421096 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.420949 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsn55\" (UniqueName: \"kubernetes.io/projected/8f682402-daa7-4b27-ad69-3edcc02524db-kube-api-access-jsn55\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.421096 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.421007 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-console-config\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.421096 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.421031 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-oauth-config\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.421096 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.421057 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-serving-cert\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.421096 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.421093 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-service-ca\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.421261 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.421115 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-oauth-serving-cert\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.522242 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.522210 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsn55\" (UniqueName: \"kubernetes.io/projected/8f682402-daa7-4b27-ad69-3edcc02524db-kube-api-access-jsn55\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.522420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.522332 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-console-config\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.522420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.522365 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-oauth-config\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.522420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.522402 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-serving-cert\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.522556 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.522427 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-service-ca\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.522556 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.522457 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-oauth-serving-cert\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.522556 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.522486 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-trusted-ca-bundle\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.523193 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.523165 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-console-config\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.523193 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.523188 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-service-ca\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.523408 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.523386 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-trusted-ca-bundle\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.523478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.523391 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-oauth-serving-cert\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.524871 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.524852 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-serving-cert\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.525358 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.525338 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-oauth-config\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.543277 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.543220 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsn55\" (UniqueName: \"kubernetes.io/projected/8f682402-daa7-4b27-ad69-3edcc02524db-kube-api-access-jsn55\") pod \"console-6466fbd457-s4hgl\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.547316 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.547268 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:00.597800 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.597777 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:00.680371 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.680282 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:00.751721 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.751688 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6466fbd457-s4hgl"] Jul 15 21:11:00.756480 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:11:00.756453 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f682402_daa7_4b27_ad69_3edcc02524db.slice/crio-8c939c0cb1a1975d214ff116ff46dbbb29f8534b5a24ad03eb430e2f758d1a40 WatchSource:0}: Error finding container 8c939c0cb1a1975d214ff116ff46dbbb29f8534b5a24ad03eb430e2f758d1a40: Status 404 returned error can't find the container with id 8c939c0cb1a1975d214ff116ff46dbbb29f8534b5a24ad03eb430e2f758d1a40 Jul 15 21:11:00.964991 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:00.964962 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:01.504198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:01.504159 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6466fbd457-s4hgl" event={"ID":"8f682402-daa7-4b27-ad69-3edcc02524db","Type":"ContainerStarted","Data":"8c939c0cb1a1975d214ff116ff46dbbb29f8534b5a24ad03eb430e2f758d1a40"} Jul 15 21:11:03.043556 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:03.043521 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:04.266796 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.266617 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:04.362745 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.362712 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:04.478837 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.478809 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:04.516253 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.516220 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6466fbd457-s4hgl" event={"ID":"8f682402-daa7-4b27-ad69-3edcc02524db","Type":"ContainerStarted","Data":"79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06"} Jul 15 21:11:04.537113 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.537025 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6466fbd457-s4hgl" podStartSLOduration=1.407984258 podStartE2EDuration="4.537011177s" totalImagesPullingTime="3.129026919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:04.536328078 +0000 UTC m=+82.662992061" watchObservedRunningTime="2026-07-15 21:11:04.537011177 +0000 UTC m=+82.663675156" Jul 15 21:11:04.587445 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.587406 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:04.817082 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.817044 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:04.911258 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:04.911227 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:05.013215 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.013172 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:05.180898 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.180813 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:05.190713 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.190686 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fh8dd"] Jul 15 21:11:05.193182 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.193166 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.196353 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.196324 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 21:11:05.196494 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.196475 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 21:11:05.196576 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.196560 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 21:11:05.197198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.197183 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 21:11:05.197905 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.197880 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 21:11:05.198718 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.198700 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-phnng\"" Jul 15 21:11:05.198832 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.198819 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 21:11:05.261772 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.261722 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-textfile\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.261925 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.261803 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.261925 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.261831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-root\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.261925 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.261850 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-wtmp\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.261925 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.261873 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.262080 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.261930 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmw4v\" (UniqueName: \"kubernetes.io/projected/67a279c1-1a0e-41d1-9a25-e076e673d17e-kube-api-access-vmw4v\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.262080 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.262012 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-sys\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.262080 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.262055 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67a279c1-1a0e-41d1-9a25-e076e673d17e-metrics-client-ca\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.262165 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.262120 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-tls\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363282 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363242 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-tls\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363294 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-textfile\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363345 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363372 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-root\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363396 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-wtmp\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363461 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-root\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363515 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363529 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-wtmp\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363583 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw4v\" (UniqueName: \"kubernetes.io/projected/67a279c1-1a0e-41d1-9a25-e076e673d17e-kube-api-access-vmw4v\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363613 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-sys\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.363706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363649 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67a279c1-1a0e-41d1-9a25-e076e673d17e-metrics-client-ca\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.364148 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363741 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-textfile\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.364148 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.363818 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67a279c1-1a0e-41d1-9a25-e076e673d17e-sys\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.364148 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.364070 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-accelerators-collector-config\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.364236 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.364185 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/67a279c1-1a0e-41d1-9a25-e076e673d17e-metrics-client-ca\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.365647 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.365626 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-tls\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.365794 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.365739 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/67a279c1-1a0e-41d1-9a25-e076e673d17e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.374047 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.374025 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw4v\" (UniqueName: \"kubernetes.io/projected/67a279c1-1a0e-41d1-9a25-e076e673d17e-kube-api-access-vmw4v\") pod \"node-exporter-fh8dd\" (UID: \"67a279c1-1a0e-41d1-9a25-e076e673d17e\") " pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.426142 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.426113 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:05.504401 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.504314 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fh8dd" Jul 15 21:11:05.514262 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:11:05.514231 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a279c1_1a0e_41d1_9a25_e076e673d17e.slice/crio-0afb366a8f34928ae70a1113b548d209623aec37e31df6948c4b5d73bc0a53fd WatchSource:0}: Error finding container 0afb366a8f34928ae70a1113b548d209623aec37e31df6948c4b5d73bc0a53fd: Status 404 returned error can't find the container with id 0afb366a8f34928ae70a1113b548d209623aec37e31df6948c4b5d73bc0a53fd Jul 15 21:11:05.520486 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.520447 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8dd" event={"ID":"67a279c1-1a0e-41d1-9a25-e076e673d17e","Type":"ContainerStarted","Data":"0afb366a8f34928ae70a1113b548d209623aec37e31df6948c4b5d73bc0a53fd"} Jul 15 21:11:05.520856 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.520829 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:05.669417 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:05.669382 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:06.084077 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.084041 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:06.185793 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.185564 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:11:06.186018 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.185985 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:06.188790 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.188754 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191425 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191441 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191451 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191456 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191492 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191618 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191651 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v5md2\"" Jul 15 21:11:06.191876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191665 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 21:11:06.192463 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191891 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 21:11:06.192463 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.191960 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 21:11:06.205133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.205110 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:11:06.271693 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271657 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-out\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.271901 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271708 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.271901 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271777 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.271901 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271829 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.271901 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271861 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.271901 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271897 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.272166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271917 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8kht\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-kube-api-access-v8kht\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.272166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.271976 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.272166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.272008 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.272166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.272036 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.272166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.272083 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.272166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.272111 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-web-config\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.272166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.272148 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.299245 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.299179 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:06.372544 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372517 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372568 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372597 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-web-config\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372624 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372652 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-out\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372697 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372730 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372784 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372819 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372863 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.372893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372888 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8kht\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-kube-api-access-v8kht\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.373374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372923 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.373374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.372947 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.373374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.373312 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.373521 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:11:06.373406 2553 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 15 21:11:06.373521 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:11:06.373466 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls podName:c1b68f4f-92c4-429e-845b-b2aec22b0f9e nodeName:}" failed. No retries permitted until 2026-07-15 21:11:06.873445257 +0000 UTC m=+85.000109238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e") : secret "alertmanager-main-tls" not found Jul 15 21:11:06.373632 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.373563 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.373632 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.373588 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.375846 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.375818 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-web-config\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.375957 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.375898 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.376059 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.376035 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-out\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.376310 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.376285 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.376416 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.376336 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.376467 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.376411 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.376607 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.376588 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-volume\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.376991 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.376975 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.382255 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.382238 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8kht\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-kube-api-access-v8kht\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.399959 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.399930 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:06.525142 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.525114 2553 generic.go:356] "Generic (PLEG): container finished" podID="67a279c1-1a0e-41d1-9a25-e076e673d17e" containerID="d09614af8651e6ed8c37dd9c1984e6894ef374571f376d50cd55ac2c72e5f8f6" exitCode=0 Jul 15 21:11:06.525281 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.525161 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8dd" event={"ID":"67a279c1-1a0e-41d1-9a25-e076e673d17e","Type":"ContainerDied","Data":"d09614af8651e6ed8c37dd9c1984e6894ef374571f376d50cd55ac2c72e5f8f6"} Jul 15 21:11:06.858231 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.858149 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:06.877216 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.877185 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:06.879466 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:06.879446 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:07.099499 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.099461 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:11:07.235075 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.235049 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:11:07.235768 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:11:07.235733 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b68f4f_92c4_429e_845b_b2aec22b0f9e.slice/crio-48c5a0e841fcdf9f018cf4d523f28f95059190276aa952e8f0148ce30b682142 WatchSource:0}: Error finding container 48c5a0e841fcdf9f018cf4d523f28f95059190276aa952e8f0148ce30b682142: Status 404 returned error can't find the container with id 48c5a0e841fcdf9f018cf4d523f28f95059190276aa952e8f0148ce30b682142 Jul 15 21:11:07.530353 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.530271 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8dd" event={"ID":"67a279c1-1a0e-41d1-9a25-e076e673d17e","Type":"ContainerStarted","Data":"b686af78efb3ea225b93f3fba04bb14b0b47f013998b817545507422051ada78"} Jul 15 21:11:07.530353 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.530330 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fh8dd" event={"ID":"67a279c1-1a0e-41d1-9a25-e076e673d17e","Type":"ContainerStarted","Data":"05671040ea4b730d55cf8384213a9a4b6e47a397dbee01de3bbf87bc8d34eb97"} Jul 15 21:11:07.531488 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.531461 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerStarted","Data":"48c5a0e841fcdf9f018cf4d523f28f95059190276aa952e8f0148ce30b682142"} Jul 15 21:11:07.561630 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.561582 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fh8dd" podStartSLOduration=1.666055172 podStartE2EDuration="2.561567824s" totalImagesPullingTime="832.225439ms" totalInitContainerRuntime="63.287213ms" isStatefulPod=true podCreationTimestamp="2026-07-15 21:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:07.561289446 +0000 UTC m=+85.687953425" watchObservedRunningTime="2026-07-15 21:11:07.561567824 +0000 UTC m=+85.688231804" Jul 15 21:11:07.786180 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.786101 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:07.849254 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:07.849176 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:08.063047 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:08.063013 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:08.159256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:08.159215 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:08.379137 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:08.379107 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:08.438699 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:08.438666 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:08.536588 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:08.536555 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerID="e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa" exitCode=0 Jul 15 21:11:08.537082 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:08.536650 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa"} Jul 15 21:11:09.531512 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.531479 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6f5744bf5d-jnc9x"] Jul 15 21:11:09.534143 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.534113 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.539695 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.539507 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 21:11:09.539695 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.539542 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 21:11:09.539695 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.539551 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-71hqt8m9j6t86\"" Jul 15 21:11:09.539695 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.539581 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 21:11:09.539695 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.539607 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lgxdz\"" Jul 15 21:11:09.539695 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.539508 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 21:11:09.547005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.546979 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f5744bf5d-jnc9x"] Jul 15 21:11:09.598551 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.598509 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-secret-metrics-server-tls\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.598776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.598571 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.598776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.598602 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-metrics-server-audit-profiles\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.598776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.598638 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-secret-metrics-server-client-certs\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.598776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.598662 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-client-ca-bundle\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.598776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.598713 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-audit-log\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.598967 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.598780 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbf7\" (UniqueName: \"kubernetes.io/projected/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-kube-api-access-xnbf7\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699179 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-secret-metrics-server-tls\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699399 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699235 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699399 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699259 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-metrics-server-audit-profiles\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699523 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699454 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-secret-metrics-server-client-certs\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699523 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699499 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-client-ca-bundle\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699618 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699535 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-audit-log\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699939 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699712 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbf7\" (UniqueName: \"kubernetes.io/projected/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-kube-api-access-xnbf7\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.699939 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.699913 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-audit-log\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.700144 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.700122 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.700397 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.700374 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-metrics-server-audit-profiles\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.702054 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.702033 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-client-ca-bundle\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.702293 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.702272 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-secret-metrics-server-tls\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.702359 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.702312 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-secret-metrics-server-client-certs\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.708923 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.708898 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbf7\" (UniqueName: \"kubernetes.io/projected/8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330-kube-api-access-xnbf7\") pod \"metrics-server-6f5744bf5d-jnc9x\" (UID: \"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330\") " pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.844716 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.844672 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:09.857498 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.857454 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:09.858207 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:09.858184 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6466fbd457-s4hgl"] Jul 15 21:11:10.029931 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.029901 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:10.059160 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.059137 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6f5744bf5d-jnc9x"] Jul 15 21:11:10.068782 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:11:10.068734 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbf6d1c_e8d4_47c1_88c8_ae2dabf2e330.slice/crio-926a267b09d3ff950d5a473c5a53cfa3b36b32742bb9da924311f6f80f42b6a4 WatchSource:0}: Error finding container 926a267b09d3ff950d5a473c5a53cfa3b36b32742bb9da924311f6f80f42b6a4: Status 404 returned error can't find the container with id 926a267b09d3ff950d5a473c5a53cfa3b36b32742bb9da924311f6f80f42b6a4 Jul 15 21:11:10.144719 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.144672 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:10.344930 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.344895 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:10.444989 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.444898 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:10.546030 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.545987 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" event={"ID":"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330","Type":"ContainerStarted","Data":"926a267b09d3ff950d5a473c5a53cfa3b36b32742bb9da924311f6f80f42b6a4"} Jul 15 21:11:10.549152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.549129 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerStarted","Data":"fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137"} Jul 15 21:11:10.549265 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.549194 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerStarted","Data":"e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11"} Jul 15 21:11:10.549265 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.549221 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerStarted","Data":"6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39"} Jul 15 21:11:10.549265 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.549244 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerStarted","Data":"594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e"} Jul 15 21:11:10.549358 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.549275 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerStarted","Data":"090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd"} Jul 15 21:11:10.598406 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.598367 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:10.718221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.718146 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:10.814210 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:10.814175 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:11.386515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.386477 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:11.395773 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.395723 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:11:11.399321 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.399295 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.404003 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.403974 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 21:11:11.404003 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404000 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 21:11:11.404226 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404014 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 21:11:11.404226 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404026 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 21:11:11.404226 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404057 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 21:11:11.404226 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404116 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 21:11:11.404422 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404239 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 21:11:11.404499 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404482 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 21:11:11.405201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.404928 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 21:11:11.405201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.405009 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 21:11:11.405201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.405056 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bf4mmfuhfjokq\"" Jul 15 21:11:11.405201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.405168 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 21:11:11.405457 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.405216 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k8jzh\"" Jul 15 21:11:11.405657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.405635 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 21:11:11.407663 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.407637 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 21:11:11.411537 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.411514 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 21:11:11.418091 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.418061 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:11:11.498515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.498467 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:11.515630 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515590 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.515834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515645 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.515834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515674 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.515834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515706 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.515834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515732 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.515834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515780 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.515834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515807 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.515834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515855 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515880 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515903 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl5vh\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-kube-api-access-cl5vh\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.515938 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516016 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516057 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516082 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-config\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516136 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516111 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516170 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516202 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516259 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.516456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.516301 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.557115 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.557051 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerStarted","Data":"6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571"} Jul 15 21:11:11.582072 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.582027 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:11.590383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.590329 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.517491098 podStartE2EDuration="5.590309101s" totalImagesPullingTime="3.055797988s" totalInitContainerRuntime="17.020015ms" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:11.58931812 +0000 UTC m=+89.715982102" watchObservedRunningTime="2026-07-15 21:11:11.590309101 +0000 UTC m=+89.716973073" Jul 15 21:11:11.617576 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617544 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617601 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617623 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617674 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617690 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617706 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617725 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617739 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl5vh\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-kube-api-access-cl5vh\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.617789 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617795 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617824 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617845 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617865 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-config\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617897 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617966 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.617993 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.618025 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.618116 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.618666 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.618250 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.622937 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.620184 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.623166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.623143 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.625456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.625426 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.628615 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.627476 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.628615 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.628276 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.628793 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.628712 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.629036 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.629007 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-config\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.629526 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.629499 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.629902 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.629878 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.631952 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.630239 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.631952 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.630512 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.631952 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.631399 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.633160 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.632397 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.633160 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.632812 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.633319 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.633196 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.633411 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.633389 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.633889 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.633841 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.634918 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.634875 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.635172 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.635155 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.643830 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.643743 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl5vh\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-kube-api-access-cl5vh\") pod \"prometheus-k8s-0\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.680515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.680475 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:11.711573 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.711524 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:11.739688 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.739651 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:11.854445 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:11.854412 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:11:11.857974 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:11:11.857929 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4861372c_4581_437b_a0c7_5459707120b9.slice/crio-938d824016a46e09fc026b1c1e2b33782291c0486d0124174aebccc7635a39dd WatchSource:0}: Error finding container 938d824016a46e09fc026b1c1e2b33782291c0486d0124174aebccc7635a39dd: Status 404 returned error can't find the container with id 938d824016a46e09fc026b1c1e2b33782291c0486d0124174aebccc7635a39dd Jul 15 21:11:12.562218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:12.562174 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" event={"ID":"8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330","Type":"ContainerStarted","Data":"245edd2e3168bbc58ce8be1784288ae23a2509eabbe520555789ea218173d281"} Jul 15 21:11:12.563729 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:12.563703 2553 generic.go:356] "Generic (PLEG): container finished" podID="4861372c-4581-437b-a0c7-5459707120b9" containerID="84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f" exitCode=0 Jul 15 21:11:12.563869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:12.563789 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f"} Jul 15 21:11:12.563936 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:12.563868 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerStarted","Data":"938d824016a46e09fc026b1c1e2b33782291c0486d0124174aebccc7635a39dd"} Jul 15 21:11:12.579361 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:12.579310 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" podStartSLOduration=1.9092063160000001 podStartE2EDuration="3.579292067s" totalImagesPullingTime="1.670085751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:12.578835268 +0000 UTC m=+90.705499249" watchObservedRunningTime="2026-07-15 21:11:12.579292067 +0000 UTC m=+90.705956046" Jul 15 21:11:13.697995 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:13.697950 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:15.578195 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:15.578159 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerStarted","Data":"9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c"} Jul 15 21:11:15.578549 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:15.578229 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerStarted","Data":"e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c"} Jul 15 21:11:16.191542 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:16.191506 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:16.298335 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:16.298291 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:17.590291 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:17.590260 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerStarted","Data":"8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef"} Jul 15 21:11:17.590646 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:17.590353 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerStarted","Data":"25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb"} Jul 15 21:11:17.590646 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:17.590382 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerStarted","Data":"529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280"} Jul 15 21:11:17.769081 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:17.768994 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:17.866196 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:17.866163 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:17.967051 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:17.967003 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:18.598490 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:18.598460 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerStarted","Data":"9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a"} Jul 15 21:11:18.645744 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:18.645687 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.051214408 podStartE2EDuration="7.645671516s" totalImagesPullingTime="4.57328428s" totalInitContainerRuntime="21.172828ms" isStatefulPod=false podCreationTimestamp="2026-07-15 21:11:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:11:18.645311188 +0000 UTC m=+96.771975169" watchObservedRunningTime="2026-07-15 21:11:18.645671516 +0000 UTC m=+96.772335496" Jul 15 21:11:19.246909 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:19.246865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:19.347994 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:19.347962 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:20.807419 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:20.807379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:20.908133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:20.908099 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:20.965896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:20.965860 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:21.242550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:21.242460 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:21.301331 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:21.301300 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:21.712494 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:21.712467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:11:27.493182 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:27.493148 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-58zwt" Jul 15 21:11:29.845714 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:29.845665 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:29.846123 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:29.845817 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:34.885591 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:34.885533 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6466fbd457-s4hgl" podUID="8f682402-daa7-4b27-ad69-3edcc02524db" containerName="console" containerID="cri-o://79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06" gracePeriod=15 Jul 15 21:11:35.121730 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.121708 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:35.225385 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225301 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsn55\" (UniqueName: \"kubernetes.io/projected/8f682402-daa7-4b27-ad69-3edcc02524db-kube-api-access-jsn55\") pod \"8f682402-daa7-4b27-ad69-3edcc02524db\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " Jul 15 21:11:35.225385 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225337 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-service-ca\") pod \"8f682402-daa7-4b27-ad69-3edcc02524db\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " Jul 15 21:11:35.225385 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225374 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-console-config\") pod \"8f682402-daa7-4b27-ad69-3edcc02524db\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " Jul 15 21:11:35.225660 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225393 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-trusted-ca-bundle\") pod \"8f682402-daa7-4b27-ad69-3edcc02524db\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " Jul 15 21:11:35.225660 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225552 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-oauth-config\") pod \"8f682402-daa7-4b27-ad69-3edcc02524db\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " Jul 15 21:11:35.225660 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225589 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-serving-cert\") pod \"8f682402-daa7-4b27-ad69-3edcc02524db\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " Jul 15 21:11:35.225660 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225611 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-oauth-serving-cert\") pod \"8f682402-daa7-4b27-ad69-3edcc02524db\" (UID: \"8f682402-daa7-4b27-ad69-3edcc02524db\") " Jul 15 21:11:35.225877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225794 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f682402-daa7-4b27-ad69-3edcc02524db" (UID: "8f682402-daa7-4b27-ad69-3edcc02524db"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:11:35.225877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225805 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-console-config" (OuterVolumeSpecName: "console-config") pod "8f682402-daa7-4b27-ad69-3edcc02524db" (UID: "8f682402-daa7-4b27-ad69-3edcc02524db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:11:35.225877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225841 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f682402-daa7-4b27-ad69-3edcc02524db" (UID: "8f682402-daa7-4b27-ad69-3edcc02524db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:11:35.226007 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225950 2553 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-service-ca\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:11:35.226007 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225970 2553 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-console-config\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:11:35.226007 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.225989 2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-trusted-ca-bundle\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:11:35.226196 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.226176 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f682402-daa7-4b27-ad69-3edcc02524db" (UID: "8f682402-daa7-4b27-ad69-3edcc02524db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:11:35.227608 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.227578 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f682402-daa7-4b27-ad69-3edcc02524db-kube-api-access-jsn55" (OuterVolumeSpecName: "kube-api-access-jsn55") pod "8f682402-daa7-4b27-ad69-3edcc02524db" (UID: "8f682402-daa7-4b27-ad69-3edcc02524db"). InnerVolumeSpecName "kube-api-access-jsn55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:11:35.227707 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.227632 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f682402-daa7-4b27-ad69-3edcc02524db" (UID: "8f682402-daa7-4b27-ad69-3edcc02524db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:11:35.227707 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.227652 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f682402-daa7-4b27-ad69-3edcc02524db" (UID: "8f682402-daa7-4b27-ad69-3edcc02524db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:11:35.327060 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.327019 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsn55\" (UniqueName: \"kubernetes.io/projected/8f682402-daa7-4b27-ad69-3edcc02524db-kube-api-access-jsn55\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:11:35.327060 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.327051 2553 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-oauth-config\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:11:35.327060 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.327060 2553 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f682402-daa7-4b27-ad69-3edcc02524db-console-serving-cert\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:11:35.327060 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.327070 2553 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f682402-daa7-4b27-ad69-3edcc02524db-oauth-serving-cert\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:11:35.661441 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.661407 2553 generic.go:356] "Generic (PLEG): container finished" podID="8f682402-daa7-4b27-ad69-3edcc02524db" containerID="79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06" exitCode=0 Jul 15 21:11:35.661620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.661450 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6466fbd457-s4hgl" Jul 15 21:11:35.661620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.661493 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6466fbd457-s4hgl" event={"ID":"8f682402-daa7-4b27-ad69-3edcc02524db","Type":"ContainerDied","Data":"79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06"} Jul 15 21:11:35.661620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.661540 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6466fbd457-s4hgl" event={"ID":"8f682402-daa7-4b27-ad69-3edcc02524db","Type":"ContainerDied","Data":"8c939c0cb1a1975d214ff116ff46dbbb29f8534b5a24ad03eb430e2f758d1a40"} Jul 15 21:11:35.661620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.661564 2553 scope.go:117] "RemoveContainer" containerID="79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06" Jul 15 21:11:35.670619 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.670602 2553 scope.go:117] "RemoveContainer" containerID="79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06" Jul 15 21:11:35.670894 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:11:35.670875 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06\": container with ID starting with 79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06 not found: ID does not exist" containerID="79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06" Jul 15 21:11:35.670958 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.670901 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06"} err="failed to get container status \"79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06\": rpc error: code = NotFound desc = could not find container \"79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06\": container with ID starting with 79535008442e516f970502128e95a5af9298c04acb8c01a7e05ad6a78796bd06 not found: ID does not exist" Jul 15 21:11:35.682139 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.682117 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6466fbd457-s4hgl"] Jul 15 21:11:35.685876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:35.685856 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6466fbd457-s4hgl"] Jul 15 21:11:36.254265 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:36.254235 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:36.347437 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:36.347398 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:11:37.142602 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:37.142572 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f682402-daa7-4b27-ad69-3edcc02524db" path="/var/lib/kubelet/pods/8f682402-daa7-4b27-ad69-3edcc02524db/volumes" Jul 15 21:11:49.850814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:49.850780 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:11:49.854368 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:11:49.854343 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6f5744bf5d-jnc9x" Jul 15 21:12:01.870349 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:01.870301 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:11.712234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:11.712193 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:11.730534 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:11.730509 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:11.804026 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:11.804000 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:24.363623 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:24.363585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:24.621564 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:24.621478 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:25.427058 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.427010 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:12:25.427472 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.427375 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="alertmanager" containerID="cri-o://090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd" gracePeriod=120 Jul 15 21:12:25.427472 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.427402 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-web" containerID="cri-o://6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39" gracePeriod=120 Jul 15 21:12:25.427589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.427460 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy" containerID="cri-o://e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11" gracePeriod=120 Jul 15 21:12:25.427589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.427426 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="prom-label-proxy" containerID="cri-o://6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571" gracePeriod=120 Jul 15 21:12:25.427589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.427468 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="config-reloader" containerID="cri-o://594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e" gracePeriod=120 Jul 15 21:12:25.427589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.427402 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-metric" containerID="cri-o://fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137" gracePeriod=120 Jul 15 21:12:25.840974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.840944 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerID="6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571" exitCode=0 Jul 15 21:12:25.840974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.840967 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerID="fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137" exitCode=0 Jul 15 21:12:25.840974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.840975 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerID="e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11" exitCode=0 Jul 15 21:12:25.840974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.840982 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerID="594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e" exitCode=0 Jul 15 21:12:25.841218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.840988 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerID="090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd" exitCode=0 Jul 15 21:12:25.841218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.841014 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571"} Jul 15 21:12:25.841218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.841073 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137"} Jul 15 21:12:25.841218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.841100 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11"} Jul 15 21:12:25.841218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.841124 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e"} Jul 15 21:12:25.841218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:25.841148 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd"} Jul 15 21:12:26.669519 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.669494 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:12:26.760384 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760295 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-web-config\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760389 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-web\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760425 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-main-db\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760448 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-trusted-ca-bundle\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760476 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8kht\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-kube-api-access-v8kht\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760500 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-cluster-tls-config\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760538 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-out\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760568 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760593 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-tls-assets\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760624 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-volume\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760646 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-metrics-client-ca\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760676 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760701 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\" (UID: \"c1b68f4f-92c4-429e-845b-b2aec22b0f9e\") " Jul 15 21:12:26.760814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760788 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:12:26.761186 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.760862 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:26.761186 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.761109 2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-main-db\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.761186 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.761133 2553 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.761186 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.761130 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:26.763338 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.763301 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:26.763725 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.763691 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-kube-api-access-v8kht" (OuterVolumeSpecName: "kube-api-access-v8kht") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "kube-api-access-v8kht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:12:26.763997 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.763971 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:26.764093 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.764036 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-volume" (OuterVolumeSpecName: "config-volume") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:26.764186 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.764152 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:26.764445 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.764426 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-out" (OuterVolumeSpecName: "config-out") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:12:26.764621 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.764603 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:26.764684 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.764624 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:12:26.767671 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.767498 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:26.773430 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.773405 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-web-config" (OuterVolumeSpecName: "web-config") pod "c1b68f4f-92c4-429e-845b-b2aec22b0f9e" (UID: "c1b68f4f-92c4-429e-845b-b2aec22b0f9e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:26.847848 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.847810 2553 generic.go:356] "Generic (PLEG): container finished" podID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerID="6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39" exitCode=0 Jul 15 21:12:26.847997 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.847873 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39"} Jul 15 21:12:26.847997 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.847899 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:12:26.847997 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.847948 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c1b68f4f-92c4-429e-845b-b2aec22b0f9e","Type":"ContainerDied","Data":"48c5a0e841fcdf9f018cf4d523f28f95059190276aa952e8f0148ce30b682142"} Jul 15 21:12:26.847997 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.847997 2553 scope.go:117] "RemoveContainer" containerID="6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571" Jul 15 21:12:26.856203 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.856181 2553 scope.go:117] "RemoveContainer" containerID="fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137" Jul 15 21:12:26.862106 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862085 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-main-tls\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862112 2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-tls-assets\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862126 2553 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-volume\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862139 2553 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-metrics-client-ca\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862152 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862166 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862179 2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-web-config\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862192 2553 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862204 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8kht\" (UniqueName: \"kubernetes.io/projected/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-kube-api-access-v8kht\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862216 2553 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-cluster-tls-config\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.862455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.862228 2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c1b68f4f-92c4-429e-845b-b2aec22b0f9e-config-out\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:26.864901 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.864883 2553 scope.go:117] "RemoveContainer" containerID="e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11" Jul 15 21:12:26.871694 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.871662 2553 scope.go:117] "RemoveContainer" containerID="6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39" Jul 15 21:12:26.873908 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.873885 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:12:26.878210 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.878189 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:12:26.879133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.879115 2553 scope.go:117] "RemoveContainer" containerID="594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e" Jul 15 21:12:26.885400 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.885383 2553 scope.go:117] "RemoveContainer" containerID="090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd" Jul 15 21:12:26.891840 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.891823 2553 scope.go:117] "RemoveContainer" containerID="e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa" Jul 15 21:12:26.897935 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.897917 2553 scope.go:117] "RemoveContainer" containerID="6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571" Jul 15 21:12:26.898184 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:26.898167 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571\": container with ID starting with 6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571 not found: ID does not exist" containerID="6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571" Jul 15 21:12:26.898256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898195 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571"} err="failed to get container status \"6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571\": rpc error: code = NotFound desc = could not find container \"6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571\": container with ID starting with 6810fb5b51718c9b74fde7f035265864e8c2fec118f82d8da8e0b39e83660571 not found: ID does not exist" Jul 15 21:12:26.898256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898222 2553 scope.go:117] "RemoveContainer" containerID="fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137" Jul 15 21:12:26.898505 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:26.898488 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137\": container with ID starting with fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137 not found: ID does not exist" containerID="fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137" Jul 15 21:12:26.898542 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898511 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137"} err="failed to get container status \"fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137\": rpc error: code = NotFound desc = could not find container \"fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137\": container with ID starting with fb2ef08f583d4660b417899b5eac03e63dcc6384cc03913a49b84717b9639137 not found: ID does not exist" Jul 15 21:12:26.898542 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898532 2553 scope.go:117] "RemoveContainer" containerID="e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11" Jul 15 21:12:26.898723 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:26.898708 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11\": container with ID starting with e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11 not found: ID does not exist" containerID="e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11" Jul 15 21:12:26.898781 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898726 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11"} err="failed to get container status \"e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11\": rpc error: code = NotFound desc = could not find container \"e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11\": container with ID starting with e1431a46c43e7efd6d0d510af045522df7d1356d6f8746be8d7d0cfe576d7a11 not found: ID does not exist" Jul 15 21:12:26.898781 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898739 2553 scope.go:117] "RemoveContainer" containerID="6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39" Jul 15 21:12:26.898969 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:26.898953 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39\": container with ID starting with 6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39 not found: ID does not exist" containerID="6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39" Jul 15 21:12:26.899017 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898973 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39"} err="failed to get container status \"6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39\": rpc error: code = NotFound desc = could not find container \"6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39\": container with ID starting with 6c443b6c1d892f57bf46ac766d547d14cfa98efe04521a438b0c684b44239d39 not found: ID does not exist" Jul 15 21:12:26.899017 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.898986 2553 scope.go:117] "RemoveContainer" containerID="594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e" Jul 15 21:12:26.899168 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:26.899155 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e\": container with ID starting with 594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e not found: ID does not exist" containerID="594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e" Jul 15 21:12:26.899201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.899173 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e"} err="failed to get container status \"594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e\": rpc error: code = NotFound desc = could not find container \"594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e\": container with ID starting with 594d8eab10ab82aa179e804209ad6018405e1bef24191ae086a4749a96c7e83e not found: ID does not exist" Jul 15 21:12:26.899201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.899185 2553 scope.go:117] "RemoveContainer" containerID="090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd" Jul 15 21:12:26.899385 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:26.899367 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd\": container with ID starting with 090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd not found: ID does not exist" containerID="090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd" Jul 15 21:12:26.899422 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.899391 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd"} err="failed to get container status \"090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd\": rpc error: code = NotFound desc = could not find container \"090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd\": container with ID starting with 090218fcf37adeee9eb3eabcb18e19bb167d698d94ae5c6f4daa4befd038a4fd not found: ID does not exist" Jul 15 21:12:26.899422 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.899408 2553 scope.go:117] "RemoveContainer" containerID="e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa" Jul 15 21:12:26.899699 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:26.899682 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa\": container with ID starting with e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa not found: ID does not exist" containerID="e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa" Jul 15 21:12:26.899783 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:26.899711 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa"} err="failed to get container status \"e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa\": rpc error: code = NotFound desc = could not find container \"e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa\": container with ID starting with e7b0a0d12e4279636061a2ef3d8de2f27c7db77ba873213c00ead96337012faa not found: ID does not exist" Jul 15 21:12:27.145818 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:27.145788 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" path="/var/lib/kubelet/pods/c1b68f4f-92c4-429e-845b-b2aec22b0f9e/volumes" Jul 15 21:12:29.812851 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:29.812818 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:12:29.813221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:29.813085 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="prometheus" containerID="cri-o://e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c" gracePeriod=600 Jul 15 21:12:29.813221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:29.813129 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="thanos-sidecar" containerID="cri-o://529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280" gracePeriod=600 Jul 15 21:12:29.813221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:29.813154 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="config-reloader" containerID="cri-o://9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c" gracePeriod=600 Jul 15 21:12:29.813221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:29.813167 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-web" containerID="cri-o://25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb" gracePeriod=600 Jul 15 21:12:29.813472 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:29.813169 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a" gracePeriod=600 Jul 15 21:12:29.813472 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:29.813127 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy" containerID="cri-o://8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef" gracePeriod=600 Jul 15 21:12:30.865859 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865821 2553 generic.go:356] "Generic (PLEG): container finished" podID="4861372c-4581-437b-a0c7-5459707120b9" containerID="9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a" exitCode=0 Jul 15 21:12:30.865859 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865849 2553 generic.go:356] "Generic (PLEG): container finished" podID="4861372c-4581-437b-a0c7-5459707120b9" containerID="8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef" exitCode=0 Jul 15 21:12:30.865859 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865858 2553 generic.go:356] "Generic (PLEG): container finished" podID="4861372c-4581-437b-a0c7-5459707120b9" containerID="529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280" exitCode=0 Jul 15 21:12:30.865859 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865865 2553 generic.go:356] "Generic (PLEG): container finished" podID="4861372c-4581-437b-a0c7-5459707120b9" containerID="9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c" exitCode=0 Jul 15 21:12:30.866318 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865872 2553 generic.go:356] "Generic (PLEG): container finished" podID="4861372c-4581-437b-a0c7-5459707120b9" containerID="e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c" exitCode=0 Jul 15 21:12:30.866318 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865887 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a"} Jul 15 21:12:30.866318 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865951 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef"} Jul 15 21:12:30.866318 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865977 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280"} Jul 15 21:12:30.866318 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.865999 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c"} Jul 15 21:12:30.866318 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:30.866023 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c"} Jul 15 21:12:31.058724 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.058703 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:31.201446 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201356 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-config-out\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201446 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201397 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-grpc-tls\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201446 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201417 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cl5vh\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-kube-api-access-cl5vh\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201446 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201438 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-metrics-client-certs\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201784 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201454 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201784 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201623 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-thanos-prometheus-http-client-file\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201784 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201669 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-config\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201944 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201811 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-kube-rbac-proxy\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.201944 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201866 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-tls-assets\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.202047 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201947 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-serving-certs-ca-bundle\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.202047 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.201977 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-kubelet-serving-ca-bundle\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.202047 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202001 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-db\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.202047 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202035 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-1\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.202221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202062 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-metrics-client-ca\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.202771 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202456 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:31.202771 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202679 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-tls\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.202771 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202722 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-0\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202780 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-web-config\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202807 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202838 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-2\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202847 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202867 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-trusted-ca-bundle\") pod \"4861372c-4581-437b-a0c7-5459707120b9\" (UID: \"4861372c-4581-437b-a0c7-5459707120b9\") " Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.202945 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.203168 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.203219 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.203242 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.203258 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-metrics-client-ca\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.203387 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:31.203656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.203543 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:31.205420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.204434 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:12:31.205420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.204510 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-config" (OuterVolumeSpecName: "config") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.205420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.204561 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.205420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.205115 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:12:31.205420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.205364 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-config-out" (OuterVolumeSpecName: "config-out") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:12:31.205674 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.205650 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.205790 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.205745 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-kube-api-access-cl5vh" (OuterVolumeSpecName: "kube-api-access-cl5vh") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "kube-api-access-cl5vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:12:31.206053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.206035 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.206381 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.206349 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.206488 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.206390 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:12:31.206657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.206633 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.207361 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.207335 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.207567 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.207537 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.215041 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.215017 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-web-config" (OuterVolumeSpecName: "web-config") pod "4861372c-4581-437b-a0c7-5459707120b9" (UID: "4861372c-4581-437b-a0c7-5459707120b9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:12:31.304622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304582 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl5vh\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-kube-api-access-cl5vh\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304619 2553 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-metrics-client-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304630 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304642 2553 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304654 2553 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-config\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304663 2553 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-kube-rbac-proxy\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304671 2553 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4861372c-4581-437b-a0c7-5459707120b9-tls-assets\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304680 2553 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304689 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-db\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304698 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304706 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304715 2553 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-web-config\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304725 2553 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304733 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304742 2553 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4861372c-4581-437b-a0c7-5459707120b9-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304750 2553 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4861372c-4581-437b-a0c7-5459707120b9-config-out\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.304912 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.304783 2553 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4861372c-4581-437b-a0c7-5459707120b9-secret-grpc-tls\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:12:31.873085 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.873052 2553 generic.go:356] "Generic (PLEG): container finished" podID="4861372c-4581-437b-a0c7-5459707120b9" containerID="25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb" exitCode=0 Jul 15 21:12:31.873478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.873128 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:31.873478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.873127 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb"} Jul 15 21:12:31.873478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.873191 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4861372c-4581-437b-a0c7-5459707120b9","Type":"ContainerDied","Data":"938d824016a46e09fc026b1c1e2b33782291c0486d0124174aebccc7635a39dd"} Jul 15 21:12:31.873478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.873226 2553 scope.go:117] "RemoveContainer" containerID="9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a" Jul 15 21:12:31.880747 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.880647 2553 scope.go:117] "RemoveContainer" containerID="8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef" Jul 15 21:12:31.890587 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.890566 2553 scope.go:117] "RemoveContainer" containerID="25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb" Jul 15 21:12:31.897033 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.897014 2553 scope.go:117] "RemoveContainer" containerID="529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280" Jul 15 21:12:31.899369 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.899350 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:12:31.904977 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.904958 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:12:31.907826 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.906597 2553 scope.go:117] "RemoveContainer" containerID="9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c" Jul 15 21:12:31.913971 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.913954 2553 scope.go:117] "RemoveContainer" containerID="e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c" Jul 15 21:12:31.920920 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.920898 2553 scope.go:117] "RemoveContainer" containerID="84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f" Jul 15 21:12:31.926937 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.926920 2553 scope.go:117] "RemoveContainer" containerID="9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a" Jul 15 21:12:31.927213 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:31.927189 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a\": container with ID starting with 9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a not found: ID does not exist" containerID="9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a" Jul 15 21:12:31.927298 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.927221 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a"} err="failed to get container status \"9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a\": rpc error: code = NotFound desc = could not find container \"9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a\": container with ID starting with 9acea5a8dde7abd7dd7c96ea501a620ea4744ffccc7bd68c05744baa155e859a not found: ID does not exist" Jul 15 21:12:31.927298 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.927242 2553 scope.go:117] "RemoveContainer" containerID="8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef" Jul 15 21:12:31.927495 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:31.927479 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef\": container with ID starting with 8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef not found: ID does not exist" containerID="8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef" Jul 15 21:12:31.927535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.927503 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef"} err="failed to get container status \"8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef\": rpc error: code = NotFound desc = could not find container \"8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef\": container with ID starting with 8f64089ebcf1892df0588cbc855839d07c0d1173901bb7ffb4f6a460cbcb4bef not found: ID does not exist" Jul 15 21:12:31.927535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.927521 2553 scope.go:117] "RemoveContainer" containerID="25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb" Jul 15 21:12:31.927733 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:31.927715 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb\": container with ID starting with 25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb not found: ID does not exist" containerID="25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb" Jul 15 21:12:31.927819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.927736 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb"} err="failed to get container status \"25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb\": rpc error: code = NotFound desc = could not find container \"25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb\": container with ID starting with 25fa99da6f86477dce3cf4d9638475897dbbe4f8372ed1653e63e366d7d755eb not found: ID does not exist" Jul 15 21:12:31.927819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.927750 2553 scope.go:117] "RemoveContainer" containerID="529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280" Jul 15 21:12:31.928025 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:31.928009 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280\": container with ID starting with 529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280 not found: ID does not exist" containerID="529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280" Jul 15 21:12:31.928069 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.928028 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280"} err="failed to get container status \"529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280\": rpc error: code = NotFound desc = could not find container \"529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280\": container with ID starting with 529f6953f982b687ccdf6d27d318d21cf9a19c1ce7ed12a4826aedd3fd585280 not found: ID does not exist" Jul 15 21:12:31.928069 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.928041 2553 scope.go:117] "RemoveContainer" containerID="9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c" Jul 15 21:12:31.928230 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:31.928216 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c\": container with ID starting with 9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c not found: ID does not exist" containerID="9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c" Jul 15 21:12:31.928269 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.928231 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c"} err="failed to get container status \"9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c\": rpc error: code = NotFound desc = could not find container \"9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c\": container with ID starting with 9587abfd6e0f0b8263fb77eb3d8887cfd24f16e288c213421af5a9868787b45c not found: ID does not exist" Jul 15 21:12:31.928269 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.928242 2553 scope.go:117] "RemoveContainer" containerID="e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c" Jul 15 21:12:31.928468 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:31.928452 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c\": container with ID starting with e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c not found: ID does not exist" containerID="e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c" Jul 15 21:12:31.928505 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.928472 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c"} err="failed to get container status \"e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c\": rpc error: code = NotFound desc = could not find container \"e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c\": container with ID starting with e2fd85d442793d9282c3126607fd01f0d6a77f9ee32ed448bc6a8c15257e747c not found: ID does not exist" Jul 15 21:12:31.928505 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.928485 2553 scope.go:117] "RemoveContainer" containerID="84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f" Jul 15 21:12:31.928717 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:12:31.928703 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f\": container with ID starting with 84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f not found: ID does not exist" containerID="84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f" Jul 15 21:12:31.928773 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.928720 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f"} err="failed to get container status \"84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f\": rpc error: code = NotFound desc = could not find container \"84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f\": container with ID starting with 84e5ed45d46643a9a29d4cd62a4a0f25170ab6f4f453e857bdedde40f394f92f not found: ID does not exist" Jul 15 21:12:31.931630 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931610 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:12:31.931899 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931887 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="config-reloader" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931901 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="config-reloader" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931908 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931913 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931922 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="alertmanager" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931927 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="alertmanager" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931933 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="config-reloader" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931938 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="config-reloader" Jul 15 21:12:31.931951 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931948 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="thanos-sidecar" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931956 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="thanos-sidecar" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931966 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="init-config-reloader" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931973 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="init-config-reloader" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931979 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="prometheus" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931984 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="prometheus" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931991 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-thanos" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.931996 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-thanos" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932002 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-web" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932007 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-web" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932013 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932018 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932024 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-metric" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932030 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-metric" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932038 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="prom-label-proxy" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932045 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="prom-label-proxy" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932051 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-web" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932055 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-web" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932064 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="init-config-reloader" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932068 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="init-config-reloader" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932074 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f682402-daa7-4b27-ad69-3edcc02524db" containerName="console" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932079 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f682402-daa7-4b27-ad69-3edcc02524db" containerName="console" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932119 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-web" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932127 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932133 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-thanos" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932138 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="alertmanager" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932143 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f682402-daa7-4b27-ad69-3edcc02524db" containerName="console" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932147 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy-metric" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932153 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="config-reloader" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932159 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="config-reloader" Jul 15 21:12:31.932155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932164 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="prom-label-proxy" Jul 15 21:12:31.932916 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932170 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="prometheus" Jul 15 21:12:31.932916 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932174 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="kube-rbac-proxy-web" Jul 15 21:12:31.932916 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932180 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b68f4f-92c4-429e-845b-b2aec22b0f9e" containerName="kube-rbac-proxy" Jul 15 21:12:31.932916 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.932184 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="4861372c-4581-437b-a0c7-5459707120b9" containerName="thanos-sidecar" Jul 15 21:12:31.935917 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.935902 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:31.938478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938451 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 21:12:31.938674 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938653 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 21:12:31.938795 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938680 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 21:12:31.938795 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938706 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 21:12:31.938795 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938713 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-k8jzh\"" Jul 15 21:12:31.938795 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938753 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bf4mmfuhfjokq\"" Jul 15 21:12:31.938795 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938656 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 21:12:31.939056 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938653 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 21:12:31.939056 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.938870 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 21:12:31.939188 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.939168 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 21:12:31.939315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.939302 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 21:12:31.939356 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.939316 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 21:12:31.939666 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.939653 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 21:12:31.939869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.939852 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 21:12:31.941565 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.941544 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 21:12:31.945421 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.945404 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 21:12:31.953192 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:31.953168 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:12:32.009743 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009704 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009743 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009743 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx8qn\" (UniqueName: \"kubernetes.io/projected/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-kube-api-access-fx8qn\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009743 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009779 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009999 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009797 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009999 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009848 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009999 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009889 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009999 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009936 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009999 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009957 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.009999 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.009995 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010041 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010071 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010103 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010123 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010142 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010198 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010174 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010207 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010221 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010243 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-config\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010258 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.010355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.010272 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111589 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111626 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111647 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111666 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111685 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111710 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111745 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111792 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111817 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111841 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-config\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.111869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111860 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111881 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111910 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111934 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx8qn\" (UniqueName: \"kubernetes.io/projected/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-kube-api-access-fx8qn\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111963 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.111989 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.112011 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.112041 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.112067 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.112098 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112665 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.112458 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.112665 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.112595 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.113314 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.113287 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.115085 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.115056 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116309 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.115297 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116309 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.115527 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116309 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.115745 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116309 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.115806 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116309 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.116220 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116595 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.116490 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116595 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.116531 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116688 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.116668 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116792 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.116755 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.116907 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.116886 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.117049 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.117020 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.117185 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.117157 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.117292 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.117264 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.117440 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.117422 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-config\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.118608 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.118591 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.120365 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.120341 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx8qn\" (UniqueName: \"kubernetes.io/projected/0b4333ce-cb9f-430d-ad4b-c499a3b41f0d-kube-api-access-fx8qn\") pod \"prometheus-k8s-0\" (UID: \"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.246148 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.246048 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:32.380528 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.380498 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 21:12:32.383634 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:12:32.383610 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4333ce_cb9f_430d_ad4b_c499a3b41f0d.slice/crio-2ab10089b078ccf2c908763e9d79176811cbb5b0934aa257469894b0a2d6147c WatchSource:0}: Error finding container 2ab10089b078ccf2c908763e9d79176811cbb5b0934aa257469894b0a2d6147c: Status 404 returned error can't find the container with id 2ab10089b078ccf2c908763e9d79176811cbb5b0934aa257469894b0a2d6147c Jul 15 21:12:32.878496 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.878462 2553 generic.go:356] "Generic (PLEG): container finished" podID="0b4333ce-cb9f-430d-ad4b-c499a3b41f0d" containerID="f21229761a520c16ea8f1ea764540da6d0799303256b3c82876d663bb921e8b5" exitCode=0 Jul 15 21:12:32.878931 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.878501 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerDied","Data":"f21229761a520c16ea8f1ea764540da6d0799303256b3c82876d663bb921e8b5"} Jul 15 21:12:32.878931 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:32.878582 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerStarted","Data":"2ab10089b078ccf2c908763e9d79176811cbb5b0934aa257469894b0a2d6147c"} Jul 15 21:12:33.142750 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.142721 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4861372c-4581-437b-a0c7-5459707120b9" path="/var/lib/kubelet/pods/4861372c-4581-437b-a0c7-5459707120b9/volumes" Jul 15 21:12:33.886211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.886177 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerStarted","Data":"23dec7fa3fdf1940162a2c02b1080b1167752c50a1c9da4c5a4f71b0f9651389"} Jul 15 21:12:33.886670 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.886249 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerStarted","Data":"fd4006cffc7e0c9d3126ddaa9ac966f40923425d8356078cc5752c33efc0780a"} Jul 15 21:12:33.886670 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.886287 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerStarted","Data":"476855b2029a619b5a01515fd16a81f10113e67e34839c9e85066e24021f1a85"} Jul 15 21:12:33.886670 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.886328 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerStarted","Data":"9655df1bacd33e0bc36049be0206d2ddf536b2853e1f3554eff7a533f22aed54"} Jul 15 21:12:33.886670 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.886370 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerStarted","Data":"d51ab603d5c49bdf0f1d9d75ab8889c3210ae9009af2b9f7f8ba27607f0976b3"} Jul 15 21:12:33.886670 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.886407 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b4333ce-cb9f-430d-ad4b-c499a3b41f0d","Type":"ContainerStarted","Data":"f70105d66aabd9ecb0952f29ed8319b897839ed22fda318d3c3e0d3d4755b9aa"} Jul 15 21:12:33.918210 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:33.918160 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.899518328 podStartE2EDuration="2.918146341s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.628013ms" isStatefulPod=false podCreationTimestamp="2026-07-15 21:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:12:33.916688631 +0000 UTC m=+172.043352611" watchObservedRunningTime="2026-07-15 21:12:33.918146341 +0000 UTC m=+172.044810352" Jul 15 21:12:34.734550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:34.734517 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:34.800909 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:34.800871 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:34.868716 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:34.868679 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:35.456427 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:35.456389 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:35.557776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:35.557702 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:37.246231 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:37.246180 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:12:46.095595 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:46.095562 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:46.194169 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:46.194130 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:12:46.252286 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:12:46.252246 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:24.301123 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:24.301082 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:31.324135 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:31.324104 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:32.247121 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:32.247067 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:13:32.262129 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:32.262096 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:13:33.093314 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:33.093288 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 21:13:39.696470 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:39.696428 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:39.810928 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:39.810899 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:39.899027 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:39.898993 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:40.003585 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:40.003513 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:40.111908 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:40.111864 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:40.759542 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:40.759510 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:40.822888 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:40.822856 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:50.307036 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:50.307000 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:50.360888 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:50.360850 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:50.424053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:50.424019 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:51.615402 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:51.615363 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:13:51.669834 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:13:51.669799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:03.774560 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:03.774521 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:03.874263 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:03.874226 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:04.090487 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:04.090439 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:34.470501 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:34.470461 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:34.583961 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:34.583928 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:42.926631 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:42.926599 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:14:42.928382 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:42.928361 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:14:42.933342 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:42.933324 2553 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 21:14:56.732424 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:56.732387 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:14:58.289160 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:14:58.289123 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:00.536411 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:00.536373 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:00.636482 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:00.636448 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:01.855241 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:01.855201 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:02.003001 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:02.002970 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:02.256228 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:02.256154 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:02.621559 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:02.621526 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:04.617037 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:04.617005 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:04.869660 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:04.869576 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:07.745166 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:07.745130 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:07.826490 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:07.826452 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:11.788606 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:11.788575 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:12.665803 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:12.665745 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:12.766406 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:12.766373 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:26.323435 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:26.323404 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:26.419937 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:26.419907 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:31.969806 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:31.969748 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:32.073788 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:32.073740 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:33.902577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:33.902541 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:42.140275 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.140240 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:42.203947 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.203913 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2"] Jul 15 21:15:42.207009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.206992 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.211162 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.211128 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 21:15:42.211162 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.211128 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:15:42.211344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.211181 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z5q8z\"" Jul 15 21:15:42.211344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.211128 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 21:15:42.211344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.211194 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 21:15:42.211344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.211184 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 21:15:42.215512 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.215489 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:42.219277 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.219259 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2"] Jul 15 21:15:42.263191 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.263156 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-cert\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.263334 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.263201 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-metrics-cert\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.263334 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.263285 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvnhm\" (UniqueName: \"kubernetes.io/projected/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-kube-api-access-hvnhm\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.263441 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.263349 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-manager-config\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.363863 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.363830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvnhm\" (UniqueName: \"kubernetes.io/projected/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-kube-api-access-hvnhm\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.364005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.363881 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-manager-config\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.364005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.363914 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-cert\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.364005 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.363929 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-metrics-cert\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.366852 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.366836 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 21:15:42.366897 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.366875 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 21:15:42.366997 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.366982 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 21:15:42.373527 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.373508 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 21:15:42.375037 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.375016 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-manager-config\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.376805 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.376788 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-metrics-cert\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.376898 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.376825 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-cert\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.384153 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.384133 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:15:42.393970 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.393919 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvnhm\" (UniqueName: \"kubernetes.io/projected/bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5-kube-api-access-hvnhm\") pod \"lws-controller-manager-7cd5bf8cdd-whjf2\" (UID: \"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5\") " pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.518086 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.518060 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-z5q8z\"" Jul 15 21:15:42.526018 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.525998 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:42.647561 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.647518 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2"] Jul 15 21:15:42.649676 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:15:42.649649 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2bfb76_73b4_4801_a9cd_6ca7eba8e7c5.slice/crio-d7dd297cdc057bc7d5983d4bbb199c5454839ebdf782312c87d0b0a61c1f5553 WatchSource:0}: Error finding container d7dd297cdc057bc7d5983d4bbb199c5454839ebdf782312c87d0b0a61c1f5553: Status 404 returned error can't find the container with id d7dd297cdc057bc7d5983d4bbb199c5454839ebdf782312c87d0b0a61c1f5553 Jul 15 21:15:42.651477 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:42.651462 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:15:43.504866 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:43.504826 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" event={"ID":"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5","Type":"ContainerStarted","Data":"d7dd297cdc057bc7d5983d4bbb199c5454839ebdf782312c87d0b0a61c1f5553"} Jul 15 21:15:45.513448 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:45.513413 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" event={"ID":"bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5","Type":"ContainerStarted","Data":"456b0dcf228247e08618721a9d6cbe433d5d2fddf3f5250d6984c9909a4bd871"} Jul 15 21:15:45.513924 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:45.513536 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:15:45.530713 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:45.530661 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" podStartSLOduration=1.350857888 podStartE2EDuration="3.530647213s" totalImagesPullingTime="2.179789325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:45.52944021 +0000 UTC m=+363.656104205" watchObservedRunningTime="2026-07-15 21:15:45.530647213 +0000 UTC m=+363.657311192" Jul 15 21:15:47.338290 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:47.338235 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:47.426482 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:47.426446 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:47.948118 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:47.948082 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:48.163819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.163755 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:48.272197 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.272119 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:48.321697 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.321668 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn"] Jul 15 21:15:48.324985 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.324970 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.327569 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.327540 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 21:15:48.327569 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.327555 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 21:15:48.327941 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.327922 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-l72zx\"" Jul 15 21:15:48.328049 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.327980 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 21:15:48.328576 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.328559 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 21:15:48.351152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.351128 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn"] Jul 15 21:15:48.413031 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.413001 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5506ba67-aa68-47fb-b001-d5fc9074531b-tmp\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.413204 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.413038 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5506ba67-aa68-47fb-b001-d5fc9074531b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.413204 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.413060 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5506ba67-aa68-47fb-b001-d5fc9074531b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.413204 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.413155 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5506ba67-aa68-47fb-b001-d5fc9074531b-manifests\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.413204 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.413191 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ngz\" (UniqueName: \"kubernetes.io/projected/5506ba67-aa68-47fb-b001-d5fc9074531b-kube-api-access-58ngz\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.444183 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.444154 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:48.514277 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.514247 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5506ba67-aa68-47fb-b001-d5fc9074531b-tmp\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.514438 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.514287 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5506ba67-aa68-47fb-b001-d5fc9074531b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.514438 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.514316 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5506ba67-aa68-47fb-b001-d5fc9074531b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.514438 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.514349 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5506ba67-aa68-47fb-b001-d5fc9074531b-manifests\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.514438 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.514379 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ngz\" (UniqueName: \"kubernetes.io/projected/5506ba67-aa68-47fb-b001-d5fc9074531b-kube-api-access-58ngz\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.514737 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.514716 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5506ba67-aa68-47fb-b001-d5fc9074531b-manifests\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.516597 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.516576 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5506ba67-aa68-47fb-b001-d5fc9074531b-tmp\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.516696 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.516678 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5506ba67-aa68-47fb-b001-d5fc9074531b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.516787 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.516752 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5506ba67-aa68-47fb-b001-d5fc9074531b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.523417 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.523360 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ngz\" (UniqueName: \"kubernetes.io/projected/5506ba67-aa68-47fb-b001-d5fc9074531b-kube-api-access-58ngz\") pod \"opendatahub-operator-controller-manager-7d5d77c45b-fr6pn\" (UID: \"5506ba67-aa68-47fb-b001-d5fc9074531b\") " pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.544818 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.544774 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:48.635072 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.635049 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:48.648364 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.648334 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:48.752942 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.752905 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:48.794294 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.794268 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn"] Jul 15 21:15:48.796751 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:15:48.796717 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5506ba67_aa68_47fb_b001_d5fc9074531b.slice/crio-55f66e0836bf9c17778552cb29b2542f6532d92b9a9021302af0e94a94429733 WatchSource:0}: Error finding container 55f66e0836bf9c17778552cb29b2542f6532d92b9a9021302af0e94a94429733: Status 404 returned error can't find the container with id 55f66e0836bf9c17778552cb29b2542f6532d92b9a9021302af0e94a94429733 Jul 15 21:15:48.895819 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:48.895754 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:15:49.531247 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:49.531170 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" event={"ID":"5506ba67-aa68-47fb-b001-d5fc9074531b","Type":"ContainerStarted","Data":"55f66e0836bf9c17778552cb29b2542f6532d92b9a9021302af0e94a94429733"} Jul 15 21:15:52.544229 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:52.544197 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" event={"ID":"5506ba67-aa68-47fb-b001-d5fc9074531b","Type":"ContainerStarted","Data":"53c241a85c89c2689e0a4176adbae3a864a9740301f60843962770a5c79e6fad"} Jul 15 21:15:53.548616 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:53.548582 2553 generic.go:356] "Generic (PLEG): container finished" podID="5506ba67-aa68-47fb-b001-d5fc9074531b" containerID="53c241a85c89c2689e0a4176adbae3a864a9740301f60843962770a5c79e6fad" exitCode=0 Jul 15 21:15:53.549108 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:53.548643 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" event={"ID":"5506ba67-aa68-47fb-b001-d5fc9074531b","Type":"ContainerDied","Data":"53c241a85c89c2689e0a4176adbae3a864a9740301f60843962770a5c79e6fad"} Jul 15 21:15:54.553968 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:54.553936 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" event={"ID":"5506ba67-aa68-47fb-b001-d5fc9074531b","Type":"ContainerStarted","Data":"0500497ccf350c471f9049acf794df21447a03643237c68fc771e52b021e9788"} Jul 15 21:15:54.554421 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:54.554068 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:15:54.576051 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:54.576009 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" podStartSLOduration=2.453590055 podStartE2EDuration="6.575996191s" totalImagesPullingTime="3.117874608s" totalInitContainerRuntime="1.004531528s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:15:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:15:54.574492909 +0000 UTC m=+372.701156881" watchObservedRunningTime="2026-07-15 21:15:54.575996191 +0000 UTC m=+372.702660171" Jul 15 21:15:56.519895 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:15:56.519867 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cd5bf8cdd-whjf2" Jul 15 21:16:00.000546 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:00.000508 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:00.139248 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:00.139213 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:00.240915 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:00.240882 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:05.139550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:05.139502 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:05.208338 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:05.208299 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:05.559105 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:05.559084 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d5d77c45b-fr6pn" Jul 15 21:16:05.568824 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:05.568794 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:05.669394 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:05.669357 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:07.215165 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:07.215134 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:07.356349 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:07.356313 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:07.783263 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:07.783230 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:07.970490 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:07.970459 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:08.084698 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.084673 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf"] Jul 15 21:16:08.087013 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.086987 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.093889 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.090273 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 21:16:08.093889 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.090525 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 21:16:08.093889 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.090728 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bw8sk\"" Jul 15 21:16:08.093889 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.090971 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 21:16:08.093889 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.090997 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 21:16:08.105500 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.105473 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf"] Jul 15 21:16:08.173353 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.173325 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9085bd7-6531-4cc4-abfa-37981373f289-tls-certs\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.173462 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.173379 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq2f\" (UniqueName: \"kubernetes.io/projected/e9085bd7-6531-4cc4-abfa-37981373f289-kube-api-access-lwq2f\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.173462 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.173427 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9085bd7-6531-4cc4-abfa-37981373f289-tmp\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.211032 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.211004 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:08.274384 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.274362 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq2f\" (UniqueName: \"kubernetes.io/projected/e9085bd7-6531-4cc4-abfa-37981373f289-kube-api-access-lwq2f\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.274678 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.274392 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9085bd7-6531-4cc4-abfa-37981373f289-tmp\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.274678 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.274429 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9085bd7-6531-4cc4-abfa-37981373f289-tls-certs\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.276468 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.276444 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9085bd7-6531-4cc4-abfa-37981373f289-tmp\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.276751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.276731 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9085bd7-6531-4cc4-abfa-37981373f289-tls-certs\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.283881 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.283861 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq2f\" (UniqueName: \"kubernetes.io/projected/e9085bd7-6531-4cc4-abfa-37981373f289-kube-api-access-lwq2f\") pod \"kube-auth-proxy-5cbf464cb7-bvmzf\" (UID: \"e9085bd7-6531-4cc4-abfa-37981373f289\") " pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.297460 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.297435 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:08.396169 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.396088 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:08.400828 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.400815 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" Jul 15 21:16:08.531974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.531943 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf"] Jul 15 21:16:08.535506 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:16:08.535481 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9085bd7_6531_4cc4_abfa_37981373f289.slice/crio-0c64431991809faf006e2a4a406b235c8bb2f674e6427b6885ec4a48577894bb WatchSource:0}: Error finding container 0c64431991809faf006e2a4a406b235c8bb2f674e6427b6885ec4a48577894bb: Status 404 returned error can't find the container with id 0c64431991809faf006e2a4a406b235c8bb2f674e6427b6885ec4a48577894bb Jul 15 21:16:08.608207 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.608169 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" event={"ID":"e9085bd7-6531-4cc4-abfa-37981373f289","Type":"ContainerStarted","Data":"0c64431991809faf006e2a4a406b235c8bb2f674e6427b6885ec4a48577894bb"} Jul 15 21:16:08.942395 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:08.942368 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:09.151152 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:09.151111 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:09.247402 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:09.247284 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:09.475994 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:09.475957 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:09.551965 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:09.551884 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:10.500195 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:10.500148 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:11.620550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:11.620518 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" event={"ID":"e9085bd7-6531-4cc4-abfa-37981373f289","Type":"ContainerStarted","Data":"5261e32207abdb7a329b78f6d07760db29e8c9c601c0dc75c00304c5178942a2"} Jul 15 21:16:11.637238 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:11.637187 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cbf464cb7-bvmzf" podStartSLOduration=1.175625911 podStartE2EDuration="3.637171515s" totalImagesPullingTime="2.461545604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:16:11.636363439 +0000 UTC m=+389.763027424" watchObservedRunningTime="2026-07-15 21:16:11.637171515 +0000 UTC m=+389.763835494" Jul 15 21:16:19.690217 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:19.690181 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:19.840052 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:19.840015 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:19.926389 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:19.926352 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:20.065526 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:20.065489 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:20.224648 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:20.224613 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:20.328927 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:20.328839 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:20.533467 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:20.533436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:20.634954 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:20.634869 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:20.884726 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:20.884686 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:21.036277 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:21.036191 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:21.181804 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:21.181746 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:24.301016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:24.300982 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:24.362952 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:24.362912 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:16:58.814896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:16:58.814858 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:03.379070 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:03.379022 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:08.797183 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:08.797069 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:08.900042 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:08.899998 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:08.981299 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:08.981259 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:09.863449 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:09.863417 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:09.958363 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:09.958328 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:27.881158 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:27.881118 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:27.981585 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:27.981557 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:34.250459 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:34.250425 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:37.708746 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:37.708715 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:39.672034 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:39.672004 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:40.441686 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:40.441646 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:40.529889 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:40.529857 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:40.638873 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:40.638845 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:40.779686 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:40.779604 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:41.152211 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:41.152180 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:41.517801 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:41.517687 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:41.666251 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:41.666216 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:43.935891 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:43.935860 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:44.058008 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:44.057970 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:44.858088 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:44.858049 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:46.451024 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:46.450992 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:47.744582 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.744546 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf"] Jul 15 21:17:47.747592 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.747570 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.750708 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.750686 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hzbjk\"" Jul 15 21:17:47.750878 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.750727 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 21:17:47.751016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.750995 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 21:17:47.751087 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.751023 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 21:17:47.751087 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.751008 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 21:17:47.761983 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.761953 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf"] Jul 15 21:17:47.763933 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.763894 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:47.876108 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.876072 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8164491b-a8ff-4353-8d77-f58a36a6c4ab-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.876108 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.876122 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8164491b-a8ff-4353-8d77-f58a36a6c4ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.876374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.876151 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjxr9\" (UniqueName: \"kubernetes.io/projected/8164491b-a8ff-4353-8d77-f58a36a6c4ab-kube-api-access-tjxr9\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.977263 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.977233 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8164491b-a8ff-4353-8d77-f58a36a6c4ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.977455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.977270 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjxr9\" (UniqueName: \"kubernetes.io/projected/8164491b-a8ff-4353-8d77-f58a36a6c4ab-kube-api-access-tjxr9\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.977455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.977321 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8164491b-a8ff-4353-8d77-f58a36a6c4ab-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.977455 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:17:47.977400 2553 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 15 21:17:47.977611 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:17:47.977486 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8164491b-a8ff-4353-8d77-f58a36a6c4ab-plugin-serving-cert podName:8164491b-a8ff-4353-8d77-f58a36a6c4ab nodeName:}" failed. No retries permitted until 2026-07-15 21:17:48.477464025 +0000 UTC m=+486.604127999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/8164491b-a8ff-4353-8d77-f58a36a6c4ab-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-w6sgf" (UID: "8164491b-a8ff-4353-8d77-f58a36a6c4ab") : secret "plugin-serving-cert" not found Jul 15 21:17:47.977903 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.977885 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8164491b-a8ff-4353-8d77-f58a36a6c4ab-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:47.990278 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:47.990252 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjxr9\" (UniqueName: \"kubernetes.io/projected/8164491b-a8ff-4353-8d77-f58a36a6c4ab-kube-api-access-tjxr9\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:48.321315 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:48.321273 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:48.482173 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:48.482137 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8164491b-a8ff-4353-8d77-f58a36a6c4ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:48.484463 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:48.484433 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8164491b-a8ff-4353-8d77-f58a36a6c4ab-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w6sgf\" (UID: \"8164491b-a8ff-4353-8d77-f58a36a6c4ab\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:48.656378 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:48.656296 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" Jul 15 21:17:48.783023 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:48.782992 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf"] Jul 15 21:17:48.785911 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:17:48.785882 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8164491b_a8ff_4353_8d77_f58a36a6c4ab.slice/crio-cb51e822852041cbd41a977f43c7d8637f1edab9203d446de3d94dd52b76e545 WatchSource:0}: Error finding container cb51e822852041cbd41a977f43c7d8637f1edab9203d446de3d94dd52b76e545: Status 404 returned error can't find the container with id cb51e822852041cbd41a977f43c7d8637f1edab9203d446de3d94dd52b76e545 Jul 15 21:17:48.962616 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:48.962525 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" event={"ID":"8164491b-a8ff-4353-8d77-f58a36a6c4ab","Type":"ContainerStarted","Data":"cb51e822852041cbd41a977f43c7d8637f1edab9203d446de3d94dd52b76e545"} Jul 15 21:17:49.336660 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:49.336633 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:49.436628 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:49.436583 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:51.304459 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:51.304421 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:51.520880 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:51.520838 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:51.608653 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:51.608543 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:53.874240 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:53.874205 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:53.974307 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:53.974271 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:57.039614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:57.039578 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:57.159561 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:57.159510 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:17:58.091012 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.090982 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k"] Jul 15 21:17:58.110697 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.110671 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k"] Jul 15 21:17:58.110879 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.110796 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.113874 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.113851 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ggchg\"" Jul 15 21:17:58.270110 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.270069 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c7ad7b1-57ee-4171-b10a-30319be7184b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.270296 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.270201 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/3c7ad7b1-57ee-4171-b10a-30319be7184b-kube-api-access-ntqgq\") pod \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.370835 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.370722 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/3c7ad7b1-57ee-4171-b10a-30319be7184b-kube-api-access-ntqgq\") pod \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.370835 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.370817 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c7ad7b1-57ee-4171-b10a-30319be7184b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.371319 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.371289 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c7ad7b1-57ee-4171-b10a-30319be7184b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.384589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.384556 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/3c7ad7b1-57ee-4171-b10a-30319be7184b-kube-api-access-ntqgq\") pod \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.423362 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.423326 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:17:58.531646 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.531609 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k"] Jul 15 21:17:58.552153 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:17:58.552123 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k"] Jul 15 21:18:01.812614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:01.812573 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:01.909306 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:01.909274 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:06.471577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:06.471528 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:06.573922 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:06.573890 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:10.790061 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:10.790024 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:10.886915 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:10.886878 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:11.056400 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:11.056366 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" event={"ID":"8164491b-a8ff-4353-8d77-f58a36a6c4ab","Type":"ContainerStarted","Data":"e9a09eeb1190de276bb9b4b0560fe88cb1e9f935ad1f961363376c73ecc7ac87"} Jul 15 21:18:11.083226 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:11.083179 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w6sgf" podStartSLOduration=2.786334375 podStartE2EDuration="24.08316373s" totalImagesPullingTime="21.296829355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:18:11.08257664 +0000 UTC m=+509.209240622" watchObservedRunningTime="2026-07-15 21:18:11.08316373 +0000 UTC m=+509.209827708" Jul 15 21:18:12.446884 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:18:12.446838 2553 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-d5l7k_kuadrant-system_3c7ad7b1-57ee-4171-b10a-30319be7184b_0(161a102ec106f8e7d1c3b9966b55e6fec5bba379d8c4c6a4a1d76efa284e016e): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-d5l7k to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 15 21:18:12.447123 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:18:12.446918 2553 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-d5l7k_kuadrant-system_3c7ad7b1-57ee-4171-b10a-30319be7184b_0(161a102ec106f8e7d1c3b9966b55e6fec5bba379d8c4c6a4a1d76efa284e016e): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-d5l7k to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:13.063341 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.063312 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:13.065730 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:18:13.065703 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" is forbidden: User \"system:node:ip-10-0-136-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-220.ec2.internal' and this object" podUID="3c7ad7b1-57ee-4171-b10a-30319be7184b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:13.141253 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:18:13.141220 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" is forbidden: User \"system:node:ip-10-0-136-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-220.ec2.internal' and this object" podUID="3c7ad7b1-57ee-4171-b10a-30319be7184b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:13.191090 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.191067 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:13.193159 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:18:13.193137 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" is forbidden: User \"system:node:ip-10-0-136-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-220.ec2.internal' and this object" podUID="3c7ad7b1-57ee-4171-b10a-30319be7184b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:13.291119 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.291092 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/3c7ad7b1-57ee-4171-b10a-30319be7184b-kube-api-access-ntqgq\") pod \"3c7ad7b1-57ee-4171-b10a-30319be7184b\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " Jul 15 21:18:13.291234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.291156 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c7ad7b1-57ee-4171-b10a-30319be7184b-extensions-socket-volume\") pod \"3c7ad7b1-57ee-4171-b10a-30319be7184b\" (UID: \"3c7ad7b1-57ee-4171-b10a-30319be7184b\") " Jul 15 21:18:13.291403 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.291384 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c7ad7b1-57ee-4171-b10a-30319be7184b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3c7ad7b1-57ee-4171-b10a-30319be7184b" (UID: "3c7ad7b1-57ee-4171-b10a-30319be7184b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:18:13.292979 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.292957 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7ad7b1-57ee-4171-b10a-30319be7184b-kube-api-access-ntqgq" (OuterVolumeSpecName: "kube-api-access-ntqgq") pod "3c7ad7b1-57ee-4171-b10a-30319be7184b" (UID: "3c7ad7b1-57ee-4171-b10a-30319be7184b"). InnerVolumeSpecName "kube-api-access-ntqgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:18:13.392230 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.392177 2553 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c7ad7b1-57ee-4171-b10a-30319be7184b-extensions-socket-volume\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:18:13.392230 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:13.392197 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntqgq\" (UniqueName: \"kubernetes.io/projected/3c7ad7b1-57ee-4171-b10a-30319be7184b-kube-api-access-ntqgq\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:18:14.066369 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:14.066336 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:14.069232 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:18:14.069206 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" is forbidden: User \"system:node:ip-10-0-136-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-220.ec2.internal' and this object" podUID="3c7ad7b1-57ee-4171-b10a-30319be7184b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:14.073916 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:18:14.073888 2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-d5l7k\" is forbidden: User \"system:node:ip-10-0-136-220.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-220.ec2.internal' and this object" podUID="3c7ad7b1-57ee-4171-b10a-30319be7184b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-d5l7k" Jul 15 21:18:15.141931 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:15.141899 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c7ad7b1-57ee-4171-b10a-30319be7184b" path="/var/lib/kubelet/pods/3c7ad7b1-57ee-4171-b10a-30319be7184b/volumes" Jul 15 21:18:17.876147 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:17.876110 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:17.977279 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:17.977228 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:22.085184 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:22.085141 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:22.185061 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:22.185029 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:26.780206 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:26.780167 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:26.889962 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:26.889924 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:26.986586 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:26.986556 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:27.086470 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:27.086442 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:38.546704 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:38.546619 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:49.307989 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.307955 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:49.410795 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.410747 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:49.510880 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.510843 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:49.714357 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.714275 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:18:49.756431 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.756398 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:18:49.756431 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.756435 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:18:49.756611 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.756512 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:49.758943 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.758927 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 21:18:49.879580 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.879552 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/85a317db-7cba-4e98-935b-5353a133aaf3-config-file\") pod \"limitador-limitador-54d6d988d5-kvffp\" (UID: \"85a317db-7cba-4e98-935b-5353a133aaf3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:49.879580 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.879583 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f94sb\" (UniqueName: \"kubernetes.io/projected/85a317db-7cba-4e98-935b-5353a133aaf3-kube-api-access-f94sb\") pod \"limitador-limitador-54d6d988d5-kvffp\" (UID: \"85a317db-7cba-4e98-935b-5353a133aaf3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:49.980304 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.980239 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/85a317db-7cba-4e98-935b-5353a133aaf3-config-file\") pod \"limitador-limitador-54d6d988d5-kvffp\" (UID: \"85a317db-7cba-4e98-935b-5353a133aaf3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:49.980304 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.980270 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f94sb\" (UniqueName: \"kubernetes.io/projected/85a317db-7cba-4e98-935b-5353a133aaf3-kube-api-access-f94sb\") pod \"limitador-limitador-54d6d988d5-kvffp\" (UID: \"85a317db-7cba-4e98-935b-5353a133aaf3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:49.980886 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.980865 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/85a317db-7cba-4e98-935b-5353a133aaf3-config-file\") pod \"limitador-limitador-54d6d988d5-kvffp\" (UID: \"85a317db-7cba-4e98-935b-5353a133aaf3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:49.988933 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:49.988912 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f94sb\" (UniqueName: \"kubernetes.io/projected/85a317db-7cba-4e98-935b-5353a133aaf3-kube-api-access-f94sb\") pod \"limitador-limitador-54d6d988d5-kvffp\" (UID: \"85a317db-7cba-4e98-935b-5353a133aaf3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:50.065853 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:50.065830 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:50.160141 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:50.160099 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:50.188886 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:50.188864 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:18:50.191429 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:18:50.191400 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a317db_7cba_4e98_935b_5353a133aaf3.slice/crio-210b24c1d7f5404403d7a05c40d058003e54396b4f8f264b6064eb8453a7341e WatchSource:0}: Error finding container 210b24c1d7f5404403d7a05c40d058003e54396b4f8f264b6064eb8453a7341e: Status 404 returned error can't find the container with id 210b24c1d7f5404403d7a05c40d058003e54396b4f8f264b6064eb8453a7341e Jul 15 21:18:50.259813 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:50.259729 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:18:51.195530 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:51.195485 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" event={"ID":"85a317db-7cba-4e98-935b-5353a133aaf3","Type":"ContainerStarted","Data":"210b24c1d7f5404403d7a05c40d058003e54396b4f8f264b6064eb8453a7341e"} Jul 15 21:18:53.205200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:53.205167 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" event={"ID":"85a317db-7cba-4e98-935b-5353a133aaf3","Type":"ContainerStarted","Data":"454b394d065c84e011d049c8c7dd3ef1ca623050bb63622f733b72f74d2ee82d"} Jul 15 21:18:53.205633 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:53.205324 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:18:53.226371 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:18:53.226323 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" podStartSLOduration=1.841798094 podStartE2EDuration="4.226306179s" totalImagesPullingTime="2.384508085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:18:53.225848711 +0000 UTC m=+551.352512692" watchObservedRunningTime="2026-07-15 21:18:53.226306179 +0000 UTC m=+551.352970160" Jul 15 21:19:04.210692 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:04.210660 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-kvffp" Jul 15 21:19:10.305439 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:10.305400 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:10.408209 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:10.408179 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:10.850462 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:10.850424 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:23.029625 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:23.029539 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:23.385688 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:23.385649 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:29.826537 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:29.826502 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:29.942273 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:29.942235 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:30.048067 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:30.048030 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:30.132206 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:30.132121 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:33.256969 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:33.256934 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:33.353403 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:33.353374 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:35.683389 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:35.683357 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:19:42.954195 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:42.954169 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:19:42.954598 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:19:42.954245 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:20:05.769497 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:05.769409 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:07.261258 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:07.261233 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:07.298852 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:07.298820 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:20:17.066419 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:17.066386 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:17.156423 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:17.156364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:25.550019 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:25.549972 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:25.999405 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:25.999370 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:20:42.754662 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:42.754622 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:43.200445 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:43.200414 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:20:52.015635 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.015605 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs"] Jul 15 21:20:52.018989 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.018970 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.021291 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.021270 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 21:20:52.022069 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.022052 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 21:20:52.022172 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.022082 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2cvfv\"" Jul 15 21:20:52.022172 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.022162 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 21:20:52.029395 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.029372 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs"] Jul 15 21:20:52.044259 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.044223 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:20:52.148627 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.148591 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.148627 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.148632 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.148904 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.148676 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvkt\" (UniqueName: \"kubernetes.io/projected/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-kube-api-access-fwvkt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.148904 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.148710 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.148904 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.148821 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.148904 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.148854 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.249983 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.249959 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.249987 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.250020 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvkt\" (UniqueName: \"kubernetes.io/projected/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-kube-api-access-fwvkt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.250061 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.250114 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250342 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.250144 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250442 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.250422 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250649 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.250627 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.250704 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.250668 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.252247 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.252226 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.252353 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.252332 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.258038 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.258013 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvkt\" (UniqueName: \"kubernetes.io/projected/2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548-kube-api-access-fwvkt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgzzs\" (UID: \"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.328734 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.328708 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:20:52.449184 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.449157 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs"] Jul 15 21:20:52.451491 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:20:52.451462 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3832a1_d1c2_40bc_a9b7_85c5ef6aa548.slice/crio-3a644384e001b871061cc5b87adee028742e20b647e95f32785d1a9d0a1bc5db WatchSource:0}: Error finding container 3a644384e001b871061cc5b87adee028742e20b647e95f32785d1a9d0a1bc5db: Status 404 returned error can't find the container with id 3a644384e001b871061cc5b87adee028742e20b647e95f32785d1a9d0a1bc5db Jul 15 21:20:52.452836 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.452816 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:20:52.668175 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:52.668095 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" event={"ID":"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548","Type":"ContainerStarted","Data":"3a644384e001b871061cc5b87adee028742e20b647e95f32785d1a9d0a1bc5db"} Jul 15 21:20:53.001153 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:53.001072 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:20:53.673923 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:53.673893 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" event={"ID":"2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548","Type":"ContainerStarted","Data":"323605c58093fc65a5d979397d66f73d83eca6d26505346ba5534a7b37fa44c9"} Jul 15 21:20:53.690372 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:20:53.690326 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" podStartSLOduration=1.6689687439999998 podStartE2EDuration="2.690310559s" totalImagesPullingTime="1.021341815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:20:53.689832256 +0000 UTC m=+671.816496236" watchObservedRunningTime="2026-07-15 21:20:53.690310559 +0000 UTC m=+671.816974540" Jul 15 21:21:01.138864 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:01.138826 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:02.328833 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:02.328802 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:21:02.329244 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:02.328862 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:21:02.340717 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:02.340692 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:21:02.719298 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:02.719215 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgzzs" Jul 15 21:21:02.801883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:02.801856 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:21:04.745831 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:04.745789 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:10.894078 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:10.894045 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:21:15.291359 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.291324 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb"] Jul 15 21:21:15.294270 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.294255 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.297056 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.297037 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 15 21:21:15.327820 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.327800 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb"] Jul 15 21:21:15.341606 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.341585 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rsh\" (UniqueName: \"kubernetes.io/projected/bb856868-bee2-435f-8f1b-5f89edc26216-kube-api-access-t6rsh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.341706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.341617 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.341706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.341636 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.341876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.341725 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.341876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.341800 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.341876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.341855 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb856868-bee2-435f-8f1b-5f89edc26216-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.342509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.342490 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:21:15.443037 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443004 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443176 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443045 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443176 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443087 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443176 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443108 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443176 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443145 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb856868-bee2-435f-8f1b-5f89edc26216-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443176 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443164 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rsh\" (UniqueName: \"kubernetes.io/projected/bb856868-bee2-435f-8f1b-5f89edc26216-kube-api-access-t6rsh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443459 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443602 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443581 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.443654 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.443627 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.445178 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.445154 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb856868-bee2-435f-8f1b-5f89edc26216-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.445438 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.445421 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb856868-bee2-435f-8f1b-5f89edc26216-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.450649 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.450628 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rsh\" (UniqueName: \"kubernetes.io/projected/bb856868-bee2-435f-8f1b-5f89edc26216-kube-api-access-t6rsh\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb\" (UID: \"bb856868-bee2-435f-8f1b-5f89edc26216\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.602955 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.602928 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:15.727426 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.727402 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb"] Jul 15 21:21:15.729355 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:21:15.729321 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb856868_bee2_435f_8f1b_5f89edc26216.slice/crio-d634e7819844cbbd083d6af752c1241ba21a078a14eaa6320446af3c87ff1759 WatchSource:0}: Error finding container d634e7819844cbbd083d6af752c1241ba21a078a14eaa6320446af3c87ff1759: Status 404 returned error can't find the container with id d634e7819844cbbd083d6af752c1241ba21a078a14eaa6320446af3c87ff1759 Jul 15 21:21:15.764618 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:15.764578 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" event={"ID":"bb856868-bee2-435f-8f1b-5f89edc26216","Type":"ContainerStarted","Data":"d634e7819844cbbd083d6af752c1241ba21a078a14eaa6320446af3c87ff1759"} Jul 15 21:21:16.769710 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:16.769679 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" event={"ID":"bb856868-bee2-435f-8f1b-5f89edc26216","Type":"ContainerStarted","Data":"34c23da447cd1d24c8c4efea8af19bb83907499e20a7fa0dabebdc327ff3bc3b"} Jul 15 21:21:16.786450 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:16.786405 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" podStartSLOduration=1.616772962 podStartE2EDuration="1.786391255s" totalImagesPullingTime="169.618293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:21:16.785483736 +0000 UTC m=+694.912147715" watchObservedRunningTime="2026-07-15 21:21:16.786391255 +0000 UTC m=+694.913055234" Jul 15 21:21:18.598979 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:18.598946 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:21:25.603781 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:25.603732 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:25.604137 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:25.603811 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:25.615747 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:25.615725 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:25.815032 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:25.815000 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb" Jul 15 21:21:47.371992 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:21:47.371894 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:22:10.403566 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:22:10.403536 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:22:10.500985 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:22:10.500954 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:22:11.620483 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:22:11.620451 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:22:11.719906 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:22:11.719871 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:22:33.693639 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:22:33.693585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:22:42.008668 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:22:42.008631 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:22:52.290449 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:22:52.290409 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:23:04.507203 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:23:04.507121 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:23:12.691169 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:23:12.691135 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:23:23.189877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:23:23.189847 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:23:25.790688 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:23:25.790644 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:27.286754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:23:27.286711 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:23:49.999131 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:23:49.999098 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:24:03.194057 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:03.194021 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:24:09.083433 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:09.083393 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:09.183594 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:09.183563 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:09.284720 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:09.284685 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:15.131213 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:15.131177 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:15.211869 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:15.211830 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:15.298918 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:15.298882 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:15.399614 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:15.399531 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:15.510744 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:15.510709 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:21.280909 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:21.280865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:21.345380 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:21.345349 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:37.706797 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:37.706708 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:24:42.975696 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:42.975660 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:24:42.976559 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:42.976541 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:24:44.784883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:44.784833 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:44.847175 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:44.847134 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:24:47.695513 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:47.695475 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:24:56.599332 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:56.599300 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:24:57.060774 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:24:57.060727 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:25:09.692507 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:25:09.692476 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:25:19.724619 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:25:19.724586 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:25:28.793968 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:25:28.793933 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:25:40.389325 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:25:40.389287 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:25:49.793515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:25:49.793479 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:26:15.694658 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:26:15.694579 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:26:24.200996 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:26:24.200965 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:26:46.996122 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:26:46.996089 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:26:56.790429 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:26:56.790392 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:26:58.756797 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:26:58.756744 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:27:00.593561 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:00.593531 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:27:13.296775 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:13.296734 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:27:14.772368 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:14.772334 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:27:23.599788 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:23.599741 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:27:34.301294 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:34.301218 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:27:39.858898 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:39.858850 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:27:43.394194 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:43.394164 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:27:52.404754 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:27:52.404719 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:29:41.493818 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:29:41.493785 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:29:42.997176 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:29:42.997149 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:29:43.000483 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:29:43.000460 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:29:50.389264 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:29:50.389231 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:30:03.600557 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:30:03.600525 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:30:05.744419 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:30:05.744390 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:16.203457 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:30:16.203426 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:30:25.889679 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:30:25.889646 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:30:36.699252 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:30:36.699207 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:30:45.294664 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:30:45.294630 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:30:55.792839 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:30:55.792787 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:31:07.394183 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:31:07.394155 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:31:16.995769 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:31:16.995723 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:31:28.588356 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:31:28.588322 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:31:51.988537 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:31:51.988498 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:31:59.298567 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:31:59.298529 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:32:08.787245 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:32:08.787165 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:32:19.198986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:32:19.198952 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:32:28.205379 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:32:28.205341 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:32:40.208556 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:32:40.208521 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:32:49.195685 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:32:49.195656 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:32:58.291876 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:32:58.291841 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:33:07.899307 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:33:07.899275 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:33:16.393660 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:33:16.393624 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:33:25.492127 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:33:25.492094 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:33:30.895414 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:33:30.895385 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:33:39.498489 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:33:39.498411 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:33:48.198058 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:33:48.198018 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:33:58.404927 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:33:58.404895 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:34:10.189199 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:10.189168 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:34:21.995655 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:21.995618 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:34:32.195113 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:32.195076 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:34:43.019065 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:43.019035 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:34:43.023540 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:43.023518 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:34:43.887077 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:43.887040 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:34:55.095259 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:55.095228 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:34:58.215927 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:34:58.215896 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:03.992397 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:03.992356 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:35:06.985220 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:06.985187 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.248425 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:10.248393 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:12.392096 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:12.392068 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:16.805957 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:16.805921 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:35:20.692671 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:20.692642 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:25.874080 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:25.874045 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:26.499431 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:26.499400 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:35:29.162583 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:29.162553 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:29.332465 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:29.332429 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:29.596405 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:29.596377 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:35.999907 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:35.999871 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.099024 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:36.098995 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.284057 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:36.283992 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.402007 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:36.401978 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.544229 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:36.544156 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.699923 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:36.699895 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:36.797171 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:36.797103 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.421620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:37.421584 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:37.520057 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:37.520017 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:40.147851 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:40.147823 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:40.396674 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:40.396509 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:35:42.645851 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:42.645819 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:42.766420 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:42.766364 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:44.377417 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:44.377392 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:48.543985 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:48.543949 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:48.628090 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:48.628054 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.181782 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.181742 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:35:49.299231 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.299157 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.398959 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.398928 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.556581 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.556550 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j"] Jul 15 21:35:49.560419 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.560400 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.563782 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.563750 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d28lw\"" Jul 15 21:35:49.563977 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.563964 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-620eb2f1-serving-cert\"" Jul 15 21:35:49.564059 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.564042 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 21:35:49.564626 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.564612 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 15 21:35:49.573678 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.573657 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j"] Jul 15 21:35:49.607055 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.607029 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.698133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.698104 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee551edc-421e-4b64-ac1d-3f12088b5282-openshift-service-ca\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.698256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.698164 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxth\" (UniqueName: \"kubernetes.io/projected/ee551edc-421e-4b64-ac1d-3f12088b5282-kube-api-access-lfxth\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.698333 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.698256 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee551edc-421e-4b64-ac1d-3f12088b5282-maas-api-tls\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.792402 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.792377 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:49.798943 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.798923 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee551edc-421e-4b64-ac1d-3f12088b5282-openshift-service-ca\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.799016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.798967 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxth\" (UniqueName: \"kubernetes.io/projected/ee551edc-421e-4b64-ac1d-3f12088b5282-kube-api-access-lfxth\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.799016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.798993 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee551edc-421e-4b64-ac1d-3f12088b5282-maas-api-tls\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.799543 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.799523 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee551edc-421e-4b64-ac1d-3f12088b5282-openshift-service-ca\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.801317 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.801298 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee551edc-421e-4b64-ac1d-3f12088b5282-maas-api-tls\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.806894 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.806832 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxth\" (UniqueName: \"kubernetes.io/projected/ee551edc-421e-4b64-ac1d-3f12088b5282-kube-api-access-lfxth\") pod \"maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j\" (UID: \"ee551edc-421e-4b64-ac1d-3f12088b5282\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:49.872693 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:49.872673 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:50.006883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:50.006861 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j"] Jul 15 21:35:50.008420 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:35:50.008387 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee551edc_421e_4b64_ac1d_3f12088b5282.slice/crio-4b80390b8b4d9c9dcec405157fe8adeb6a199757a5a5d89068e2f03ad4f5b811 WatchSource:0}: Error finding container 4b80390b8b4d9c9dcec405157fe8adeb6a199757a5a5d89068e2f03ad4f5b811: Status 404 returned error can't find the container with id 4b80390b8b4d9c9dcec405157fe8adeb6a199757a5a5d89068e2f03ad4f5b811 Jul 15 21:35:50.009667 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:50.009648 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:35:50.163379 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:50.163297 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" event={"ID":"ee551edc-421e-4b64-ac1d-3f12088b5282","Type":"ContainerStarted","Data":"4b80390b8b4d9c9dcec405157fe8adeb6a199757a5a5d89068e2f03ad4f5b811"} Jul 15 21:35:50.345437 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:50.345408 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:50.448316 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:50.448240 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:52.173992 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:52.173964 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" event={"ID":"ee551edc-421e-4b64-ac1d-3f12088b5282","Type":"ContainerStarted","Data":"c49e78875d3ba33202c632290e629e8c00341ec46dc95275df932f575b3d1d56"} Jul 15 21:35:52.174368 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:52.174087 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:35:52.182368 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:52.182343 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:52.191488 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:52.191447 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" podStartSLOduration=1.148517339 podStartE2EDuration="3.191433636s" totalImagesPullingTime="2.042916297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:35:52.190802279 +0000 UTC m=+1570.317466261" watchObservedRunningTime="2026-07-15 21:35:52.191433636 +0000 UTC m=+1570.318097637" Jul 15 21:35:54.942571 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:54.942536 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:55.050900 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:55.050868 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:56.867390 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:56.867361 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:58.183331 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:35:58.183307 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j" Jul 15 21:36:01.057833 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:01.057802 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.158253 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:01.158222 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.313503 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:01.313426 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.412218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:01.412186 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.508716 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:01.508688 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:01.632907 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:01.632799 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:02.304070 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:02.304027 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:02.395252 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:02.395225 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:02.494870 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:02.494837 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:04.601501 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:04.601474 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:05.206909 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:05.206878 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:36:06.944634 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:06.944603 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:07.046773 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:07.046726 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:09.289098 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:09.289067 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:12.877091 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:12.877057 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:12.948887 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:12.948860 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:13.402898 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:13.402866 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:13.499823 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:13.499795 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:13.612012 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:13.611979 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:15.075434 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:15.075405 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:17.493077 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:17.493040 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:36:20.928410 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:20.928379 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:21.034967 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:21.034939 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:21.130201 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:21.130161 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:21.228668 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:21.228600 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:21.678883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:21.678850 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:47.702495 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:47.702463 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:54.022441 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:54.022406 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:54.137031 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:54.136994 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:54.266313 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:54.266287 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:56.723787 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:56.723745 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:56.780629 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:36:56.780604 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:05.326690 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:05.326654 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:10.177440 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:10.177412 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:16.134065 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:16.134022 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:16.242421 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:16.242387 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:16.336589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:16.336556 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:26.964457 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:26.964423 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:27.740292 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:27.740252 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:33.555456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:33.555421 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:33.656863 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:33.656834 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:33.756776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:33.756727 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:33.859089 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:33.859021 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:34.662032 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:34.661995 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:34.764004 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:34.763976 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:34.859422 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:34.859395 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:36.097479 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:36.097451 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:37:36.514605 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:36.514528 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:42.567608 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:42.567569 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:42.667295 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:42.667265 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:43.268883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:43.268852 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:43.700533 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:43.700502 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:37:49.548556 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:49.548523 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:49.648775 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:49.648742 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:49.805578 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:49.805548 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:50.200729 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:50.200651 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:50.827469 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:50.827433 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:52.529893 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.526750 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:37:52.537546 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.537505 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:52.600218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.600193 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq"] Jul 15 21:37:52.603283 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.603264 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.605654 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.605635 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c2c3a008-serving-cert\"" Jul 15 21:37:52.615095 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.615076 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq"] Jul 15 21:37:52.636008 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.635958 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:52.682277 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.682248 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50ed3-0339-4479-bca1-e59b95e48ea1-openshift-service-ca\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.682432 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.682286 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd8mh\" (UniqueName: \"kubernetes.io/projected/e9b50ed3-0339-4479-bca1-e59b95e48ea1-kube-api-access-qd8mh\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.682432 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.682348 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b50ed3-0339-4479-bca1-e59b95e48ea1-maas-api-tls\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.717022 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.716997 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:52.783484 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.783437 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50ed3-0339-4479-bca1-e59b95e48ea1-openshift-service-ca\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.783484 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.783465 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd8mh\" (UniqueName: \"kubernetes.io/projected/e9b50ed3-0339-4479-bca1-e59b95e48ea1-kube-api-access-qd8mh\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.783633 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.783492 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b50ed3-0339-4479-bca1-e59b95e48ea1-maas-api-tls\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.783633 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:37:52.783590 2553 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-serving-cert: secret "maas-api-e2e-mt-c2c3a008-serving-cert" not found Jul 15 21:37:52.783740 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:37:52.783654 2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9b50ed3-0339-4479-bca1-e59b95e48ea1-maas-api-tls podName:e9b50ed3-0339-4479-bca1-e59b95e48ea1 nodeName:}" failed. No retries permitted until 2026-07-15 21:37:53.283639368 +0000 UTC m=+1691.410303326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e9b50ed3-0339-4479-bca1-e59b95e48ea1-maas-api-tls") pod "maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" (UID: "e9b50ed3-0339-4479-bca1-e59b95e48ea1") : secret "maas-api-e2e-mt-c2c3a008-serving-cert" not found Jul 15 21:37:52.784106 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.784084 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b50ed3-0339-4479-bca1-e59b95e48ea1-openshift-service-ca\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:52.796969 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:52.796949 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd8mh\" (UniqueName: \"kubernetes.io/projected/e9b50ed3-0339-4479-bca1-e59b95e48ea1-kube-api-access-qd8mh\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:53.022657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.022619 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:53.286711 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.286681 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b50ed3-0339-4479-bca1-e59b95e48ea1-maas-api-tls\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:53.289061 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.289041 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e9b50ed3-0339-4479-bca1-e59b95e48ea1-maas-api-tls\") pod \"maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq\" (UID: \"e9b50ed3-0339-4479-bca1-e59b95e48ea1\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:53.298187 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.298156 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:53.399607 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.399578 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:53.513343 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.513320 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:53.637262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.637232 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq"] Jul 15 21:37:53.639648 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:37:53.639621 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b50ed3_0339_4479_bca1_e59b95e48ea1.slice/crio-9c793a3c15fa61cfcd141ef73af7acf50e26fa7e29089ffd9544ee9d0c2bcde6 WatchSource:0}: Error finding container 9c793a3c15fa61cfcd141ef73af7acf50e26fa7e29089ffd9544ee9d0c2bcde6: Status 404 returned error can't find the container with id 9c793a3c15fa61cfcd141ef73af7acf50e26fa7e29089ffd9544ee9d0c2bcde6 Jul 15 21:37:53.649171 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:53.649144 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" event={"ID":"e9b50ed3-0339-4479-bca1-e59b95e48ea1","Type":"ContainerStarted","Data":"9c793a3c15fa61cfcd141ef73af7acf50e26fa7e29089ffd9544ee9d0c2bcde6"} Jul 15 21:37:54.654515 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:54.654486 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" event={"ID":"e9b50ed3-0339-4479-bca1-e59b95e48ea1","Type":"ContainerStarted","Data":"c3d3ba2ec3508d2aa951a5cad7bb9d6ebfa87a3406e70901c5c2b50541eec62f"} Jul 15 21:37:54.654845 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:54.654556 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:37:54.674674 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:54.674629 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" podStartSLOduration=1.7946825039999998 podStartE2EDuration="2.67461528s" totalImagesPullingTime="879.932776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:37:54.673099209 +0000 UTC m=+1692.799763189" watchObservedRunningTime="2026-07-15 21:37:54.67461528 +0000 UTC m=+1692.801279261" Jul 15 21:37:58.048162 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:58.048124 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:58.722249 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:58.722214 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:58.822539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:58.822507 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:59.182008 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:37:59.181978 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:00.664802 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:00.664754 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq" Jul 15 21:38:03.893974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:03.893938 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:38:04.899090 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:04.899055 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:04.969549 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:04.969519 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:05.588252 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:05.588215 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:11.579885 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:11.579838 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:11.733029 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:11.733000 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:11.865053 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:11.864985 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:12.090897 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:12.090862 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:12.229048 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:12.228969 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:12.327270 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:12.327241 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:12.427366 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:12.427331 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:15.601042 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:15.601002 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:38:18.335069 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:18.335036 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:18.445339 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:18.445310 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:18.537999 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:18.537968 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:18.684299 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:18.684233 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:24.619715 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:24.619677 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:24.721742 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:24.721712 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:32.272259 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:32.272231 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:32.371563 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:32.371535 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:32.471560 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:32.471535 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.212862 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:33.212826 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.312000 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:33.311970 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.414906 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:33.414867 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:34.785493 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:34.785461 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:36.307411 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:36.307379 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:38:40.669862 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:40.669824 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:40.771134 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:40.771098 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:41.374139 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:41.374101 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:47.831882 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:47.831834 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:47.938560 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:47.938527 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:48.043038 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:48.043003 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:49.267975 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:49.267948 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:49.369279 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:49.369254 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:49.696845 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:49.696815 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:38:58.572146 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:58.572113 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:58.927658 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:38:58.927581 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:19.896478 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:19.896441 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:25.829287 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:25.829249 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:25.927800 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:25.927752 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:26.175475 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:26.175411 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:26.523303 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:26.523230 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:26.896449 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:26.896424 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:28.968449 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:28.968419 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:29.064105 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:29.064072 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:29.168605 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:29.168574 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:29.275456 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:29.275382 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:29.672848 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:29.672819 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:29.780161 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:29.780130 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:34.482389 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:34.482312 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:34.817544 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:34.817515 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:35.154656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:35.154589 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:35.950797 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:35.950745 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:37.096687 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.096657 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5"] Jul 15 21:39:37.100888 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.100862 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.103161 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.103140 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a5309f-serving-cert\"" Jul 15 21:39:37.114732 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.114710 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5"] Jul 15 21:39:37.209556 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.209528 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms95p\" (UniqueName: \"kubernetes.io/projected/b788e6c1-c6e3-47c8-b277-407731d1ce91-kube-api-access-ms95p\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.209711 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.209594 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b788e6c1-c6e3-47c8-b277-407731d1ce91-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.209711 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.209622 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b788e6c1-c6e3-47c8-b277-407731d1ce91-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.310885 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.310854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b788e6c1-c6e3-47c8-b277-407731d1ce91-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.311046 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.310900 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b788e6c1-c6e3-47c8-b277-407731d1ce91-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.311046 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.310940 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms95p\" (UniqueName: \"kubernetes.io/projected/b788e6c1-c6e3-47c8-b277-407731d1ce91-kube-api-access-ms95p\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.311575 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.311552 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b788e6c1-c6e3-47c8-b277-407731d1ce91-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.312985 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.312967 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b788e6c1-c6e3-47c8-b277-407731d1ce91-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.320074 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.320048 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms95p\" (UniqueName: \"kubernetes.io/projected/b788e6c1-c6e3-47c8-b277-407731d1ce91-kube-api-access-ms95p\") pod \"maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5\" (UID: \"b788e6c1-c6e3-47c8-b277-407731d1ce91\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.410796 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.410741 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:37.420081 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.420061 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:39:37.432322 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.432301 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:37.546558 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.546533 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5"] Jul 15 21:39:37.548680 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:39:37.548657 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb788e6c1_c6e3_47c8_b277_407731d1ce91.slice/crio-2c72ebd1b3e3468f4eeab127c0767390504b85deed9b54028f9f10f6a301bb10 WatchSource:0}: Error finding container 2c72ebd1b3e3468f4eeab127c0767390504b85deed9b54028f9f10f6a301bb10: Status 404 returned error can't find the container with id 2c72ebd1b3e3468f4eeab127c0767390504b85deed9b54028f9f10f6a301bb10 Jul 15 21:39:37.560257 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.560233 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:37.826539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.826508 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:37.924177 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:37.924156 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:38.058588 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:38.058562 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" event={"ID":"b788e6c1-c6e3-47c8-b277-407731d1ce91","Type":"ContainerStarted","Data":"e5acd9b7bf488f64458b6ca1e56c2486b9ebe65011c2a0fe8d85fa234cc6d798"} Jul 15 21:39:38.058739 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:38.058608 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" event={"ID":"b788e6c1-c6e3-47c8-b277-407731d1ce91","Type":"ContainerStarted","Data":"2c72ebd1b3e3468f4eeab127c0767390504b85deed9b54028f9f10f6a301bb10"} Jul 15 21:39:38.058739 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:38.058648 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:38.096991 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:38.096917 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" podStartSLOduration=0.789261148 podStartE2EDuration="1.096900908s" totalImagesPullingTime="307.63976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:39:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:39:38.095593676 +0000 UTC m=+1796.222257656" watchObservedRunningTime="2026-07-15 21:39:38.096900908 +0000 UTC m=+1796.223564889" Jul 15 21:39:42.587439 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:42.587406 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:43.046048 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:43.045963 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:39:43.051701 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:43.051681 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:39:44.068497 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:44.068467 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5" Jul 15 21:39:50.584254 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:50.584215 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:50.684559 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:50.684521 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:51.493402 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:51.493360 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:39:52.716000 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.715969 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b"] Jul 15 21:39:52.719153 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.719130 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.722911 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.722888 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ec9ec\"/\"test-model-4ec9ec-kserve-self-signed-certs\"" Jul 15 21:39:52.723072 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.722911 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ec9ec\"/\"kube-root-ca.crt\"" Jul 15 21:39:52.723188 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.722939 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4ec9ec\"/\"openshift-service-ca.crt\"" Jul 15 21:39:52.723188 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.722940 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4ec9ec\"/\"default-dockercfg-tbw6g\"" Jul 15 21:39:52.729972 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.729952 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b"] Jul 15 21:39:52.740916 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.740893 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:52.838006 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.837973 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-home\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.838006 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.838006 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-tmp-dir\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.838234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.838029 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-model-cache\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.838234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.838105 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45p4s\" (UniqueName: \"kubernetes.io/projected/30853933-3fa0-480b-9968-672c60c3475a-kube-api-access-45p4s\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.838234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.838130 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30853933-3fa0-480b-9968-672c60c3475a-tls-certs\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.838234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.838205 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-dshm\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939046 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939013 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45p4s\" (UniqueName: \"kubernetes.io/projected/30853933-3fa0-480b-9968-672c60c3475a-kube-api-access-45p4s\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939051 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30853933-3fa0-480b-9968-672c60c3475a-tls-certs\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939079 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-dshm\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939120 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-home\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939135 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-tmp-dir\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939157 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-model-cache\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939597 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939564 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-home\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939597 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939581 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-model-cache\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.939814 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.939615 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-tmp-dir\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.941312 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.941292 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-dshm\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.941590 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.941571 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30853933-3fa0-480b-9968-672c60c3475a-tls-certs\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:52.949402 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:52.949383 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45p4s\" (UniqueName: \"kubernetes.io/projected/30853933-3fa0-480b-9968-672c60c3475a-kube-api-access-45p4s\") pod \"test-model-4ec9ec-kserve-6fcbb464bc-4sq7b\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:53.028463 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:53.028384 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:39:53.157790 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:53.157746 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b"] Jul 15 21:39:53.160502 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:39:53.160471 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30853933_3fa0_480b_9968_672c60c3475a.slice/crio-3ad4a0582f3d676d011b8d531e819f3b729e1b90d9181af8a23fb7d0c84a8882 WatchSource:0}: Error finding container 3ad4a0582f3d676d011b8d531e819f3b729e1b90d9181af8a23fb7d0c84a8882: Status 404 returned error can't find the container with id 3ad4a0582f3d676d011b8d531e819f3b729e1b90d9181af8a23fb7d0c84a8882 Jul 15 21:39:54.128664 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:54.128627 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" event={"ID":"30853933-3fa0-480b-9968-672c60c3475a","Type":"ContainerStarted","Data":"19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1"} Jul 15 21:39:54.128664 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:54.128666 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" event={"ID":"30853933-3fa0-480b-9968-672c60c3475a","Type":"ContainerStarted","Data":"3ad4a0582f3d676d011b8d531e819f3b729e1b90d9181af8a23fb7d0c84a8882"} Jul 15 21:39:54.148631 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:54.148591 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" podStartSLOduration=2.148576575 podStartE2EDuration="2.148576575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:39:54.146899792 +0000 UTC m=+1812.273563782" watchObservedRunningTime="2026-07-15 21:39:54.148576575 +0000 UTC m=+1812.275240555" Jul 15 21:39:58.937915 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:58.937876 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:59.037920 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:39:59.037887 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:03.028807 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:03.028751 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:40:03.029208 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:03.028832 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:40:03.040532 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:03.040499 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:40:03.182509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:03.182475 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:40:07.110336 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:07.110295 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:40:17.828833 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.828798 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c"] Jul 15 21:40:17.831954 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.831936 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.834957 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.834929 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5309f\"/\"kube-root-ca.crt\"" Jul 15 21:40:17.835981 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.835962 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5309f\"/\"test-model-a5309f-kserve-self-signed-certs\"" Jul 15 21:40:17.836094 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.836021 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5309f\"/\"openshift-service-ca.crt\"" Jul 15 21:40:17.836094 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.836067 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5309f\"/\"default-dockercfg-qrsp9\"" Jul 15 21:40:17.842805 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.842774 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c"] Jul 15 21:40:17.862548 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.862516 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2nq\" (UniqueName: \"kubernetes.io/projected/cd18d395-c82c-45e9-b75d-6518ae40c7a2-kube-api-access-ct2nq\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.862548 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.862560 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tmp-dir\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.862854 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.862632 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tls-certs\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.862854 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.862655 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-dshm\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.862854 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.862691 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-model-cache\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.862854 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.862717 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-home\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.916286 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.916243 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:17.963412 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.963378 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tls-certs\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.963412 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.963417 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-dshm\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.963615 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.963451 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-model-cache\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.963615 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.963474 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-home\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.963615 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.963529 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2nq\" (UniqueName: \"kubernetes.io/projected/cd18d395-c82c-45e9-b75d-6518ae40c7a2-kube-api-access-ct2nq\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.963615 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.963559 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tmp-dir\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.964072 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.964046 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-model-cache\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.964180 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.964070 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tmp-dir\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.964180 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.964109 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-home\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.965919 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.965885 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-dshm\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.966081 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.966064 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tls-certs\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:17.975244 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:17.975213 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2nq\" (UniqueName: \"kubernetes.io/projected/cd18d395-c82c-45e9-b75d-6518ae40c7a2-kube-api-access-ct2nq\") pod \"test-model-a5309f-kserve-58b8b8866f-z972c\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:18.142875 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:18.142751 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:18.291357 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:18.291324 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c"] Jul 15 21:40:18.291707 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:40:18.291676 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd18d395_c82c_45e9_b75d_6518ae40c7a2.slice/crio-1597381ab83b61092c5da3c2a0f5cc1ec8d3c15275ab2a207088a3ce240e0eec WatchSource:0}: Error finding container 1597381ab83b61092c5da3c2a0f5cc1ec8d3c15275ab2a207088a3ce240e0eec: Status 404 returned error can't find the container with id 1597381ab83b61092c5da3c2a0f5cc1ec8d3c15275ab2a207088a3ce240e0eec Jul 15 21:40:19.234359 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:19.234314 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" event={"ID":"cd18d395-c82c-45e9-b75d-6518ae40c7a2","Type":"ContainerStarted","Data":"b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947"} Jul 15 21:40:19.234359 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:19.234363 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" event={"ID":"cd18d395-c82c-45e9-b75d-6518ae40c7a2","Type":"ContainerStarted","Data":"1597381ab83b61092c5da3c2a0f5cc1ec8d3c15275ab2a207088a3ce240e0eec"} Jul 15 21:40:19.252218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:19.252166 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" podStartSLOduration=2.252151057 podStartE2EDuration="2.252151057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:40:19.25093661 +0000 UTC m=+1837.377600589" watchObservedRunningTime="2026-07-15 21:40:19.252151057 +0000 UTC m=+1837.378815036" Jul 15 21:40:23.296288 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:23.296253 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:40:28.143617 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:28.143579 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:28.144019 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:28.143633 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:28.155303 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:28.155280 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:28.283553 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:28.283526 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:36.094009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:36.093961 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b"] Jul 15 21:40:36.116610 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:36.116581 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:36.980650 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:36.980618 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c"] Jul 15 21:40:36.993225 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:36.993198 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:37.503301 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:37.503270 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:42.574547 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:42.574513 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:42.697141 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:42.697107 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:40:51.113607 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.113547 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" podUID="30853933-3fa0-480b-9968-672c60c3475a" containerName="main" containerID="cri-o://19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1" gracePeriod=45 Jul 15 21:40:51.347971 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.347945 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:40:51.367979 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.367902 2553 generic.go:356] "Generic (PLEG): container finished" podID="30853933-3fa0-480b-9968-672c60c3475a" containerID="19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1" exitCode=0 Jul 15 21:40:51.367979 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.367950 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" Jul 15 21:40:51.368174 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.367992 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" event={"ID":"30853933-3fa0-480b-9968-672c60c3475a","Type":"ContainerDied","Data":"19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1"} Jul 15 21:40:51.368174 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.368040 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b" event={"ID":"30853933-3fa0-480b-9968-672c60c3475a","Type":"ContainerDied","Data":"3ad4a0582f3d676d011b8d531e819f3b729e1b90d9181af8a23fb7d0c84a8882"} Jul 15 21:40:51.368174 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.368064 2553 scope.go:117] "RemoveContainer" containerID="19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1" Jul 15 21:40:51.375962 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.375943 2553 scope.go:117] "RemoveContainer" containerID="19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1" Jul 15 21:40:51.376238 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:40:51.376208 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1\": container with ID starting with 19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1 not found: ID does not exist" containerID="19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1" Jul 15 21:40:51.376301 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.376249 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1"} err="failed to get container status \"19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1\": rpc error: code = NotFound desc = could not find container \"19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1\": container with ID starting with 19a5f93bee3cdaefa2ddaef7e32df8ff82b21ba9ab361a35217b4757a87b9cc1 not found: ID does not exist" Jul 15 21:40:51.460657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.460621 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-home\") pod \"30853933-3fa0-480b-9968-672c60c3475a\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " Jul 15 21:40:51.460863 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.460710 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30853933-3fa0-480b-9968-672c60c3475a-tls-certs\") pod \"30853933-3fa0-480b-9968-672c60c3475a\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " Jul 15 21:40:51.460863 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.460785 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-dshm\") pod \"30853933-3fa0-480b-9968-672c60c3475a\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " Jul 15 21:40:51.461000 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.460892 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-model-cache\") pod \"30853933-3fa0-480b-9968-672c60c3475a\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " Jul 15 21:40:51.461000 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.460931 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-home" (OuterVolumeSpecName: "home") pod "30853933-3fa0-480b-9968-672c60c3475a" (UID: "30853933-3fa0-480b-9968-672c60c3475a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:51.461000 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.460952 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-tmp-dir\") pod \"30853933-3fa0-480b-9968-672c60c3475a\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " Jul 15 21:40:51.461000 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.460994 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45p4s\" (UniqueName: \"kubernetes.io/projected/30853933-3fa0-480b-9968-672c60c3475a-kube-api-access-45p4s\") pod \"30853933-3fa0-480b-9968-672c60c3475a\" (UID: \"30853933-3fa0-480b-9968-672c60c3475a\") " Jul 15 21:40:51.461179 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.461156 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30853933-3fa0-480b-9968-672c60c3475a" (UID: "30853933-3fa0-480b-9968-672c60c3475a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:51.461179 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.461165 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-model-cache" (OuterVolumeSpecName: "model-cache") pod "30853933-3fa0-480b-9968-672c60c3475a" (UID: "30853933-3fa0-480b-9968-672c60c3475a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:51.461372 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.461350 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:51.461434 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.461377 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:51.461434 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.461390 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:51.462997 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.462971 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-dshm" (OuterVolumeSpecName: "dshm") pod "30853933-3fa0-480b-9968-672c60c3475a" (UID: "30853933-3fa0-480b-9968-672c60c3475a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:51.463091 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.463044 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30853933-3fa0-480b-9968-672c60c3475a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30853933-3fa0-480b-9968-672c60c3475a" (UID: "30853933-3fa0-480b-9968-672c60c3475a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:40:51.463091 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.463077 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30853933-3fa0-480b-9968-672c60c3475a-kube-api-access-45p4s" (OuterVolumeSpecName: "kube-api-access-45p4s") pod "30853933-3fa0-480b-9968-672c60c3475a" (UID: "30853933-3fa0-480b-9968-672c60c3475a"). InnerVolumeSpecName "kube-api-access-45p4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:40:51.562079 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.562046 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30853933-3fa0-480b-9968-672c60c3475a-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:51.562079 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.562076 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45p4s\" (UniqueName: \"kubernetes.io/projected/30853933-3fa0-480b-9968-672c60c3475a-kube-api-access-45p4s\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:51.562272 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.562089 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30853933-3fa0-480b-9968-672c60c3475a-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:51.688046 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.688019 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b"] Jul 15 21:40:51.690503 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:51.690482 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4ec9ec/test-model-4ec9ec-kserve-6fcbb464bc-4sq7b"] Jul 15 21:40:52.001354 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.001266 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" podUID="cd18d395-c82c-45e9-b75d-6518ae40c7a2" containerName="main" containerID="cri-o://b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947" gracePeriod=45 Jul 15 21:40:52.242652 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.242627 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:52.370874 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.370848 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-dshm\") pod \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " Jul 15 21:40:52.370874 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.370879 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tls-certs\") pod \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " Jul 15 21:40:52.371126 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.370924 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-model-cache\") pod \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " Jul 15 21:40:52.371126 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.371044 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct2nq\" (UniqueName: \"kubernetes.io/projected/cd18d395-c82c-45e9-b75d-6518ae40c7a2-kube-api-access-ct2nq\") pod \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " Jul 15 21:40:52.371126 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.371068 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tmp-dir\") pod \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " Jul 15 21:40:52.371126 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.371095 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-home\") pod \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\" (UID: \"cd18d395-c82c-45e9-b75d-6518ae40c7a2\") " Jul 15 21:40:52.371332 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.371247 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd18d395-c82c-45e9-b75d-6518ae40c7a2" (UID: "cd18d395-c82c-45e9-b75d-6518ae40c7a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:52.371387 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.371370 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:52.371542 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.371515 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd18d395-c82c-45e9-b75d-6518ae40c7a2" (UID: "cd18d395-c82c-45e9-b75d-6518ae40c7a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:52.371542 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.371532 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-home" (OuterVolumeSpecName: "home") pod "cd18d395-c82c-45e9-b75d-6518ae40c7a2" (UID: "cd18d395-c82c-45e9-b75d-6518ae40c7a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:52.373212 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.373184 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-dshm" (OuterVolumeSpecName: "dshm") pod "cd18d395-c82c-45e9-b75d-6518ae40c7a2" (UID: "cd18d395-c82c-45e9-b75d-6518ae40c7a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:40:52.373322 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.373267 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd18d395-c82c-45e9-b75d-6518ae40c7a2-kube-api-access-ct2nq" (OuterVolumeSpecName: "kube-api-access-ct2nq") pod "cd18d395-c82c-45e9-b75d-6518ae40c7a2" (UID: "cd18d395-c82c-45e9-b75d-6518ae40c7a2"). InnerVolumeSpecName "kube-api-access-ct2nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:40:52.373371 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.373316 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd18d395-c82c-45e9-b75d-6518ae40c7a2" (UID: "cd18d395-c82c-45e9-b75d-6518ae40c7a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:40:52.375704 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.375682 2553 generic.go:356] "Generic (PLEG): container finished" podID="cd18d395-c82c-45e9-b75d-6518ae40c7a2" containerID="b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947" exitCode=0 Jul 15 21:40:52.375807 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.375722 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" event={"ID":"cd18d395-c82c-45e9-b75d-6518ae40c7a2","Type":"ContainerDied","Data":"b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947"} Jul 15 21:40:52.375807 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.375728 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" Jul 15 21:40:52.375807 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.375777 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c" event={"ID":"cd18d395-c82c-45e9-b75d-6518ae40c7a2","Type":"ContainerDied","Data":"1597381ab83b61092c5da3c2a0f5cc1ec8d3c15275ab2a207088a3ce240e0eec"} Jul 15 21:40:52.375807 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.375787 2553 scope.go:117] "RemoveContainer" containerID="b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947" Jul 15 21:40:52.389985 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.389960 2553 scope.go:117] "RemoveContainer" containerID="b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947" Jul 15 21:40:52.390259 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:40:52.390240 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947\": container with ID starting with b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947 not found: ID does not exist" containerID="b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947" Jul 15 21:40:52.390323 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.390272 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947"} err="failed to get container status \"b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947\": rpc error: code = NotFound desc = could not find container \"b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947\": container with ID starting with b3b725a299bd23e97de9b16bf5babf770a374e0777370226068710d48ffa1947 not found: ID does not exist" Jul 15 21:40:52.400697 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.400675 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c"] Jul 15 21:40:52.410245 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.410221 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a5309f/test-model-a5309f-kserve-58b8b8866f-z972c"] Jul 15 21:40:52.472055 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.472031 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct2nq\" (UniqueName: \"kubernetes.io/projected/cd18d395-c82c-45e9-b75d-6518ae40c7a2-kube-api-access-ct2nq\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:52.472180 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.472058 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:52.472180 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.472084 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:52.472180 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.472094 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd18d395-c82c-45e9-b75d-6518ae40c7a2-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:52.472180 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:52.472104 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd18d395-c82c-45e9-b75d-6518ae40c7a2-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:40:53.143287 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:53.143252 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30853933-3fa0-480b-9968-672c60c3475a" path="/var/lib/kubelet/pods/30853933-3fa0-480b-9968-672c60c3475a/volumes" Jul 15 21:40:53.143684 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:40:53.143668 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd18d395-c82c-45e9-b75d-6518ae40c7a2" path="/var/lib/kubelet/pods/cd18d395-c82c-45e9-b75d-6518ae40c7a2/volumes" Jul 15 21:41:19.479678 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:19.479607 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:19.771453 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:19.771377 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:20.018769 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:20.018715 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:20.528523 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:20.528476 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:23.823297 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:23.823264 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:41:26.732141 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:26.732109 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:26.807411 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:26.807384 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:27.037383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:27.037307 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:27.252984 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:27.252957 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:27.638741 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:27.638713 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:28.307090 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:28.307059 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:28.402492 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:28.402463 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:29.619024 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.618987 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:29.719344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.719313 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:29.831127 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.831098 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf"] Jul 15 21:41:29.831416 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.831404 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd18d395-c82c-45e9-b75d-6518ae40c7a2" containerName="main" Jul 15 21:41:29.831475 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.831417 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd18d395-c82c-45e9-b75d-6518ae40c7a2" containerName="main" Jul 15 21:41:29.831475 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.831425 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30853933-3fa0-480b-9968-672c60c3475a" containerName="main" Jul 15 21:41:29.831475 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.831431 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="30853933-3fa0-480b-9968-672c60c3475a" containerName="main" Jul 15 21:41:29.831577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.831488 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd18d395-c82c-45e9-b75d-6518ae40c7a2" containerName="main" Jul 15 21:41:29.831577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.831497 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="30853933-3fa0-480b-9968-672c60c3475a" containerName="main" Jul 15 21:41:29.834472 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.834455 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:29.838555 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.838535 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-599e95-serving-cert\"" Jul 15 21:41:29.848181 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.848155 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf"] Jul 15 21:41:29.869872 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.869815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:29.972928 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.972905 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:29.976991 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.976973 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-maas-api-tls\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:29.977085 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.977030 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppk44\" (UniqueName: \"kubernetes.io/projected/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-kube-api-access-ppk44\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:29.977202 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:29.977113 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-openshift-service-ca\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.078326 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.078304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppk44\" (UniqueName: \"kubernetes.io/projected/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-kube-api-access-ppk44\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.078440 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.078343 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-openshift-service-ca\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.078440 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.078399 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-maas-api-tls\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.079041 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.079023 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-openshift-service-ca\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.080653 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.080632 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-maas-api-tls\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.088532 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.088508 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppk44\" (UniqueName: \"kubernetes.io/projected/b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57-kube-api-access-ppk44\") pod \"maas-api-e2e-api-a-599e95-67c64789c8-24brf\" (UID: \"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.144224 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.144171 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:30.267696 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.267661 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf"] Jul 15 21:41:30.270565 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:41:30.270540 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b19be8_6bbc_4dbb_b8c4_23c05b12ef57.slice/crio-40c261eae35844c59edd4a27dbbfe8f1db882c64e16ad93cd58b3c6a44e0fe0b WatchSource:0}: Error finding container 40c261eae35844c59edd4a27dbbfe8f1db882c64e16ad93cd58b3c6a44e0fe0b: Status 404 returned error can't find the container with id 40c261eae35844c59edd4a27dbbfe8f1db882c64e16ad93cd58b3c6a44e0fe0b Jul 15 21:41:30.271830 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.271815 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:41:30.373168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.373137 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:30.515262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.515160 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:30.532619 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:30.532590 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" event={"ID":"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57","Type":"ContainerStarted","Data":"40c261eae35844c59edd4a27dbbfe8f1db882c64e16ad93cd58b3c6a44e0fe0b"} Jul 15 21:41:31.538351 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:31.538312 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" event={"ID":"b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57","Type":"ContainerStarted","Data":"5202af1c9466fa1fd1b9ab32e2bf656b5cd0125bc3e995c2f8a84f035ad6bec6"} Jul 15 21:41:31.538701 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:31.538379 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:31.556535 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:31.556485 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" podStartSLOduration=2.01337868 podStartE2EDuration="2.556471707s" totalImagesPullingTime="543.093027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:41:31.555067142 +0000 UTC m=+1909.681731121" watchObservedRunningTime="2026-07-15 21:41:31.556471707 +0000 UTC m=+1909.683135687" Jul 15 21:41:35.150010 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:35.149971 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:35.645589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:35.645560 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:35.982952 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:35.982880 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:36.123371 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:36.123340 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:36.194126 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:36.194093 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:41:37.547854 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.547826 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-599e95-67c64789c8-24brf" Jul 15 21:41:37.618387 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.618360 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:37.689415 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.689382 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:37.818684 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.818653 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf"] Jul 15 21:41:37.822494 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.822476 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:37.824833 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.824811 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d3a40f-serving-cert\"" Jul 15 21:41:37.831443 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.831378 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf"] Jul 15 21:41:37.860883 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.860857 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:37.941235 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.941212 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86daee49-c31a-4efd-ba30-978782d82dc3-maas-api-tls\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:37.941344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.941256 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntdr7\" (UniqueName: \"kubernetes.io/projected/86daee49-c31a-4efd-ba30-978782d82dc3-kube-api-access-ntdr7\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:37.941344 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:37.941326 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86daee49-c31a-4efd-ba30-978782d82dc3-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.042007 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.041979 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86daee49-c31a-4efd-ba30-978782d82dc3-maas-api-tls\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.042104 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.042020 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntdr7\" (UniqueName: \"kubernetes.io/projected/86daee49-c31a-4efd-ba30-978782d82dc3-kube-api-access-ntdr7\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.042160 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.042141 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86daee49-c31a-4efd-ba30-978782d82dc3-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.042672 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.042656 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86daee49-c31a-4efd-ba30-978782d82dc3-openshift-service-ca\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.044249 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.044234 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86daee49-c31a-4efd-ba30-978782d82dc3-maas-api-tls\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.050626 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.050603 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntdr7\" (UniqueName: \"kubernetes.io/projected/86daee49-c31a-4efd-ba30-978782d82dc3-kube-api-access-ntdr7\") pod \"maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf\" (UID: \"86daee49-c31a-4efd-ba30-978782d82dc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.070995 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.070932 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:38.134142 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.134115 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:38.265849 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.265824 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf"] Jul 15 21:41:38.267983 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:41:38.267953 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86daee49_c31a_4efd_ba30_978782d82dc3.slice/crio-6c0d8742a86ac3e86d2de825a8455986f68f7dd55cb4115d57eba13badc9c2d2 WatchSource:0}: Error finding container 6c0d8742a86ac3e86d2de825a8455986f68f7dd55cb4115d57eba13badc9c2d2: Status 404 returned error can't find the container with id 6c0d8742a86ac3e86d2de825a8455986f68f7dd55cb4115d57eba13badc9c2d2 Jul 15 21:41:38.497094 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.497017 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:38.569985 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.569935 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" event={"ID":"86daee49-c31a-4efd-ba30-978782d82dc3","Type":"ContainerStarted","Data":"6c0d8742a86ac3e86d2de825a8455986f68f7dd55cb4115d57eba13badc9c2d2"} Jul 15 21:41:38.597235 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:38.597211 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:39.575580 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:39.575542 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" event={"ID":"86daee49-c31a-4efd-ba30-978782d82dc3","Type":"ContainerStarted","Data":"71639486f6a9247a4e255f62953a6b41cede906a2d6b38c915839eb925df59d1"} Jul 15 21:41:39.575964 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:39.575678 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:39.597697 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:39.597640 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" podStartSLOduration=2.063859937 podStartE2EDuration="2.597623665s" totalImagesPullingTime="533.763728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:41:39.596632886 +0000 UTC m=+1917.723296866" watchObservedRunningTime="2026-07-15 21:41:39.597623665 +0000 UTC m=+1917.724287645" Jul 15 21:41:43.144586 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:43.144555 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:45.584780 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:45.584740 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf" Jul 15 21:41:51.423823 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:51.423780 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:51.523193 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:51.523165 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:51.708102 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:51.708036 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:54.413399 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:54.413366 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:41:57.755643 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:57.755608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:59.346598 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:59.346558 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:59.448499 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:41:59.448470 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:03.487525 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:03.487490 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:03.589244 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:03.589210 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:03.787936 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:03.787856 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:04.023234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:04.023199 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:04.508133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:04.508097 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:06.905722 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:06.905691 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:42:10.456921 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:10.456887 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:16.225365 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:16.225331 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:16.298181 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:16.298151 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:16.569769 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:16.569741 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:16.811305 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:16.811275 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:17.162091 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:17.162060 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:17.504694 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:17.504618 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:17.619672 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:17.619638 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:19.247932 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:19.247901 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:19.354496 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:19.354462 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:19.442204 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:19.442174 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:19.632070 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:19.632036 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:19.980334 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:19.980259 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:20.084092 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:20.084067 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:20.315980 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:20.315951 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:42:24.762608 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:24.762574 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:25.109384 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:25.109354 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:25.440592 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:25.440518 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:25.719467 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:25.719378 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:27.906973 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:27.906938 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:28.005292 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:28.005254 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:28.074394 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:28.074360 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:28.202888 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:28.202811 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:28.615405 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:28.615369 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:28.715370 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:28.715344 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:33.434140 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:33.434102 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:35.796509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:35.796422 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:42:40.843462 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:40.843429 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:40.942902 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:40.942873 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:49.127680 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.127643 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd"] Jul 15 21:42:49.130984 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.130967 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.133679 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.133660 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c664d0\"/\"kube-root-ca.crt\"" Jul 15 21:42:49.134643 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.134617 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c664d0\"/\"auth-test-model-c664d0-kserve-self-signed-certs\"" Jul 15 21:42:49.134643 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.134641 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c664d0\"/\"openshift-service-ca.crt\"" Jul 15 21:42:49.134833 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.134617 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c664d0\"/\"default-dockercfg-ldnw2\"" Jul 15 21:42:49.143755 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.143735 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd"] Jul 15 21:42:49.205816 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.205790 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:49.226824 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.226802 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tls-certs\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.226900 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.226831 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-home\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.226900 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.226886 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-model-cache\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.226973 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.226909 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tmp-dir\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.226973 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.226926 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-dshm\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.226973 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.226960 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thpvw\" (UniqueName: \"kubernetes.io/projected/aed8d1d8-cc09-486f-ae44-be90e46cad6d-kube-api-access-thpvw\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.327682 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.327658 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-model-cache\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.327818 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.327687 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tmp-dir\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.327818 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.327704 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-dshm\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.327818 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.327718 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thpvw\" (UniqueName: \"kubernetes.io/projected/aed8d1d8-cc09-486f-ae44-be90e46cad6d-kube-api-access-thpvw\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.327974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.327824 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tls-certs\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.327974 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.327947 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-home\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.328130 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.328109 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tmp-dir\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.328130 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.328124 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-model-cache\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.328223 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.328187 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-home\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.330058 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.330035 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-dshm\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.330256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.330241 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tls-certs\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.338516 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.338496 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thpvw\" (UniqueName: \"kubernetes.io/projected/aed8d1d8-cc09-486f-ae44-be90e46cad6d-kube-api-access-thpvw\") pod \"auth-test-model-c664d0-kserve-bfffcd875-6l6pd\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.440184 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.440127 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:49.563439 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.563404 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd"] Jul 15 21:42:49.565570 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:42:49.565545 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaed8d1d8_cc09_486f_ae44_be90e46cad6d.slice/crio-d14f3ae886735fb9112b1af7f8fcdb5e7fadc308e7197eca63da5e3cfc4f0642 WatchSource:0}: Error finding container d14f3ae886735fb9112b1af7f8fcdb5e7fadc308e7197eca63da5e3cfc4f0642: Status 404 returned error can't find the container with id d14f3ae886735fb9112b1af7f8fcdb5e7fadc308e7197eca63da5e3cfc4f0642 Jul 15 21:42:49.594433 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.594409 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:49.699971 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.699889 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:49.876039 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.875998 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" event={"ID":"aed8d1d8-cc09-486f-ae44-be90e46cad6d","Type":"ContainerStarted","Data":"b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398"} Jul 15 21:42:49.876195 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.876048 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" event={"ID":"aed8d1d8-cc09-486f-ae44-be90e46cad6d","Type":"ContainerStarted","Data":"d14f3ae886735fb9112b1af7f8fcdb5e7fadc308e7197eca63da5e3cfc4f0642"} Jul 15 21:42:49.892566 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:49.892525 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" podStartSLOduration=0.892511305 podStartE2EDuration="892.511305ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:42:49.891617504 +0000 UTC m=+1988.018281484" watchObservedRunningTime="2026-07-15 21:42:49.892511305 +0000 UTC m=+1988.019175283" Jul 15 21:42:53.103720 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:53.103688 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:42:59.440995 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:59.440948 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:59.441427 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:59.441038 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:59.453009 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:59.452990 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:42:59.934437 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:42:59.934412 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:43:17.243808 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:17.243773 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:25.520148 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.520110 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb"] Jul 15 21:43:25.523609 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.523588 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.527163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.527139 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"openshift-service-ca.crt\"" Jul 15 21:43:25.527163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.527150 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"default-dockercfg-8smwz\"" Jul 15 21:43:25.527362 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.527153 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"auth-test-model-700cbf-kserve-self-signed-certs\"" Jul 15 21:43:25.527362 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.527205 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"kube-root-ca.crt\"" Jul 15 21:43:25.532567 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.532549 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb"] Jul 15 21:43:25.617774 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.617728 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:25.649938 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.649909 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-model-cache\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.650095 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.649943 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-home\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.650095 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.649986 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2217f545-ab39-4f75-9f77-1fdd62d281d8-tls-certs\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.650095 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.650076 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-tmp-dir\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.650221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.650114 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw574\" (UniqueName: \"kubernetes.io/projected/2217f545-ab39-4f75-9f77-1fdd62d281d8-kube-api-access-bw574\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.650221 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.650179 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-dshm\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751302 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751268 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-tmp-dir\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751302 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751304 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw574\" (UniqueName: \"kubernetes.io/projected/2217f545-ab39-4f75-9f77-1fdd62d281d8-kube-api-access-bw574\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751331 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-dshm\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751368 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-model-cache\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751391 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-home\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751539 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751447 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2217f545-ab39-4f75-9f77-1fdd62d281d8-tls-certs\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751733 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751709 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-tmp-dir\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751812 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751793 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-model-cache\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.751902 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.751879 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-home\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.753632 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.753611 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-dshm\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.753829 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.753813 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2217f545-ab39-4f75-9f77-1fdd62d281d8-tls-certs\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.759098 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.759074 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw574\" (UniqueName: \"kubernetes.io/projected/2217f545-ab39-4f75-9f77-1fdd62d281d8-kube-api-access-bw574\") pod \"auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.833806 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.833778 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:25.958838 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:25.958813 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb"] Jul 15 21:43:25.961355 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:43:25.961329 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2217f545_ab39_4f75_9f77_1fdd62d281d8.slice/crio-bbf0597b854a4fed00400c0c42cca002c3dfc3fad139affea350931c390d617d WatchSource:0}: Error finding container bbf0597b854a4fed00400c0c42cca002c3dfc3fad139affea350931c390d617d: Status 404 returned error can't find the container with id bbf0597b854a4fed00400c0c42cca002c3dfc3fad139affea350931c390d617d Jul 15 21:43:26.032125 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:26.032093 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" event={"ID":"2217f545-ab39-4f75-9f77-1fdd62d281d8","Type":"ContainerStarted","Data":"561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610"} Jul 15 21:43:26.032220 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:26.032141 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" event={"ID":"2217f545-ab39-4f75-9f77-1fdd62d281d8","Type":"ContainerStarted","Data":"bbf0597b854a4fed00400c0c42cca002c3dfc3fad139affea350931c390d617d"} Jul 15 21:43:26.050155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:26.050109 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" podStartSLOduration=1.050094471 podStartE2EDuration="1.050094471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:43:26.048388573 +0000 UTC m=+2024.175052552" watchObservedRunningTime="2026-07-15 21:43:26.050094471 +0000 UTC m=+2024.176758450" Jul 15 21:43:29.396078 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:29.396046 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:43:35.834063 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:35.834027 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:35.834653 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:35.834079 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:35.845925 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:35.845901 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:36.092810 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:36.092720 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:43:40.611236 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:40.611202 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:43:50.165405 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:50.165377 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:50.266620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:50.266588 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:51.066896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:51.066865 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:56.580383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:56.580346 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:56.636210 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:56.636179 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:59.008631 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:43:59.008594 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:44:17.501624 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:44:17.501547 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:44:37.409572 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:44:37.407086 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:44:42.350256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:44:42.350221 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:44:43.074377 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:44:43.074344 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:44:43.082680 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:44:43.082660 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:44:49.523460 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:44:49.523423 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:44:57.804294 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:44:57.804261 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:45:16.905776 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:16.905727 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:45:31.957393 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:31.957363 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd"] Jul 15 21:45:31.981484 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:31.981457 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:32.855388 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:32.855356 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb"] Jul 15 21:45:33.656711 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:33.656677 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:35.804310 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:35.804274 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:45:40.327542 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.327507 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp"] Jul 15 21:45:40.330986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.330966 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.333222 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.333199 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c664d0\"/\"sub-test-model-c664d0-kserve-self-signed-certs\"" Jul 15 21:45:40.340406 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.340389 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp"] Jul 15 21:45:40.351349 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.351323 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:40.470013 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.469982 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-home\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.470205 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.470042 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tls-certs\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.470205 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.470063 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-model-cache\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.470205 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.470132 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-dshm\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.470205 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.470167 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzst7\" (UniqueName: \"kubernetes.io/projected/c0983b6e-fd95-42a8-a27d-b80a1669beb4-kube-api-access-tzst7\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.470353 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.470251 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tmp-dir\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571266 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571241 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-dshm\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571266 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571266 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzst7\" (UniqueName: \"kubernetes.io/projected/c0983b6e-fd95-42a8-a27d-b80a1669beb4-kube-api-access-tzst7\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571468 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571303 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tmp-dir\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571468 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571421 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-home\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571564 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571509 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tls-certs\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571564 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571539 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-model-cache\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571654 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571609 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tmp-dir\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571822 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571802 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-home\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.571885 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.571871 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-model-cache\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.573419 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.573399 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-dshm\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.573735 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.573716 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tls-certs\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.580072 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.580015 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzst7\" (UniqueName: \"kubernetes.io/projected/c0983b6e-fd95-42a8-a27d-b80a1669beb4-kube-api-access-tzst7\") pod \"sub-test-model-c664d0-kserve-68b45d686f-xcmlp\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.639565 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.639542 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:40.763742 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:40.763719 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp"] Jul 15 21:45:40.765902 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:45:40.765876 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0983b6e_fd95_42a8_a27d_b80a1669beb4.slice/crio-dba227260bfced2e2cfb786e7b6b0dcba97e31b939f0a36adca2f6728da2cdbd WatchSource:0}: Error finding container dba227260bfced2e2cfb786e7b6b0dcba97e31b939f0a36adca2f6728da2cdbd: Status 404 returned error can't find the container with id dba227260bfced2e2cfb786e7b6b0dcba97e31b939f0a36adca2f6728da2cdbd Jul 15 21:45:41.611967 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:41.611923 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" event={"ID":"c0983b6e-fd95-42a8-a27d-b80a1669beb4","Type":"ContainerStarted","Data":"9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82"} Jul 15 21:45:41.612327 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:41.611986 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" event={"ID":"c0983b6e-fd95-42a8-a27d-b80a1669beb4","Type":"ContainerStarted","Data":"dba227260bfced2e2cfb786e7b6b0dcba97e31b939f0a36adca2f6728da2cdbd"} Jul 15 21:45:41.631341 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:41.631285 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" podStartSLOduration=1.631266702 podStartE2EDuration="1.631266702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:45:41.630984583 +0000 UTC m=+2159.757648563" watchObservedRunningTime="2026-07-15 21:45:41.631266702 +0000 UTC m=+2159.757930683" Jul 15 21:45:46.976666 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:46.976616 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" podUID="aed8d1d8-cc09-486f-ae44-be90e46cad6d" containerName="main" containerID="cri-o://b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398" gracePeriod=45 Jul 15 21:45:47.219167 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.219144 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:45:47.335397 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335368 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thpvw\" (UniqueName: \"kubernetes.io/projected/aed8d1d8-cc09-486f-ae44-be90e46cad6d-kube-api-access-thpvw\") pod \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " Jul 15 21:45:47.335550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335445 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tmp-dir\") pod \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " Jul 15 21:45:47.335550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335469 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tls-certs\") pod \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " Jul 15 21:45:47.335550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335484 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-home\") pod \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " Jul 15 21:45:47.335550 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335524 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-model-cache\") pod \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " Jul 15 21:45:47.335725 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335558 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-dshm\") pod \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\" (UID: \"aed8d1d8-cc09-486f-ae44-be90e46cad6d\") " Jul 15 21:45:47.335826 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335789 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aed8d1d8-cc09-486f-ae44-be90e46cad6d" (UID: "aed8d1d8-cc09-486f-ae44-be90e46cad6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:47.335826 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335800 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-home" (OuterVolumeSpecName: "home") pod "aed8d1d8-cc09-486f-ae44-be90e46cad6d" (UID: "aed8d1d8-cc09-486f-ae44-be90e46cad6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:47.335826 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.335820 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "aed8d1d8-cc09-486f-ae44-be90e46cad6d" (UID: "aed8d1d8-cc09-486f-ae44-be90e46cad6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:47.337634 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.337603 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-dshm" (OuterVolumeSpecName: "dshm") pod "aed8d1d8-cc09-486f-ae44-be90e46cad6d" (UID: "aed8d1d8-cc09-486f-ae44-be90e46cad6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:47.337716 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.337708 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aed8d1d8-cc09-486f-ae44-be90e46cad6d" (UID: "aed8d1d8-cc09-486f-ae44-be90e46cad6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:45:47.337779 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.337713 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed8d1d8-cc09-486f-ae44-be90e46cad6d-kube-api-access-thpvw" (OuterVolumeSpecName: "kube-api-access-thpvw") pod "aed8d1d8-cc09-486f-ae44-be90e46cad6d" (UID: "aed8d1d8-cc09-486f-ae44-be90e46cad6d"). InnerVolumeSpecName "kube-api-access-thpvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:45:47.436661 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.436636 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thpvw\" (UniqueName: \"kubernetes.io/projected/aed8d1d8-cc09-486f-ae44-be90e46cad6d-kube-api-access-thpvw\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:47.436661 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.436661 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:47.436877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.436675 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aed8d1d8-cc09-486f-ae44-be90e46cad6d-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:47.436877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.436686 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:47.436877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.436697 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:47.436877 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.436708 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aed8d1d8-cc09-486f-ae44-be90e46cad6d-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:47.640311 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.640227 2553 generic.go:356] "Generic (PLEG): container finished" podID="aed8d1d8-cc09-486f-ae44-be90e46cad6d" containerID="b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398" exitCode=0 Jul 15 21:45:47.640311 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.640263 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" event={"ID":"aed8d1d8-cc09-486f-ae44-be90e46cad6d","Type":"ContainerDied","Data":"b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398"} Jul 15 21:45:47.640311 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.640306 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" event={"ID":"aed8d1d8-cc09-486f-ae44-be90e46cad6d","Type":"ContainerDied","Data":"d14f3ae886735fb9112b1af7f8fcdb5e7fadc308e7197eca63da5e3cfc4f0642"} Jul 15 21:45:47.640564 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.640277 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd" Jul 15 21:45:47.640564 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.640322 2553 scope.go:117] "RemoveContainer" containerID="b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398" Jul 15 21:45:47.649433 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.649418 2553 scope.go:117] "RemoveContainer" containerID="b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398" Jul 15 21:45:47.649675 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:45:47.649658 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398\": container with ID starting with b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398 not found: ID does not exist" containerID="b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398" Jul 15 21:45:47.649727 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.649691 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398"} err="failed to get container status \"b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398\": rpc error: code = NotFound desc = could not find container \"b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398\": container with ID starting with b5f509d300b92d296b66f2eb6438ea532804cf60afb4caa8978e9913b7584398 not found: ID does not exist" Jul 15 21:45:47.661684 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.661659 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd"] Jul 15 21:45:47.667622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.667602 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/auth-test-model-c664d0-kserve-bfffcd875-6l6pd"] Jul 15 21:45:47.874418 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:47.874369 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" podUID="2217f545-ab39-4f75-9f77-1fdd62d281d8" containerName="main" containerID="cri-o://561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610" gracePeriod=45 Jul 15 21:45:48.115236 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.115216 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:45:48.245023 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.244950 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2217f545-ab39-4f75-9f77-1fdd62d281d8-tls-certs\") pod \"2217f545-ab39-4f75-9f77-1fdd62d281d8\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " Jul 15 21:45:48.245023 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245015 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-home\") pod \"2217f545-ab39-4f75-9f77-1fdd62d281d8\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " Jul 15 21:45:48.245234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245030 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-dshm\") pod \"2217f545-ab39-4f75-9f77-1fdd62d281d8\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " Jul 15 21:45:48.245234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245045 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-tmp-dir\") pod \"2217f545-ab39-4f75-9f77-1fdd62d281d8\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " Jul 15 21:45:48.245234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245087 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-model-cache\") pod \"2217f545-ab39-4f75-9f77-1fdd62d281d8\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " Jul 15 21:45:48.245234 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245108 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw574\" (UniqueName: \"kubernetes.io/projected/2217f545-ab39-4f75-9f77-1fdd62d281d8-kube-api-access-bw574\") pod \"2217f545-ab39-4f75-9f77-1fdd62d281d8\" (UID: \"2217f545-ab39-4f75-9f77-1fdd62d281d8\") " Jul 15 21:45:48.245423 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245339 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2217f545-ab39-4f75-9f77-1fdd62d281d8" (UID: "2217f545-ab39-4f75-9f77-1fdd62d281d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:48.245423 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245356 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2217f545-ab39-4f75-9f77-1fdd62d281d8" (UID: "2217f545-ab39-4f75-9f77-1fdd62d281d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:48.245423 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.245366 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-home" (OuterVolumeSpecName: "home") pod "2217f545-ab39-4f75-9f77-1fdd62d281d8" (UID: "2217f545-ab39-4f75-9f77-1fdd62d281d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:48.247258 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.247235 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2217f545-ab39-4f75-9f77-1fdd62d281d8-kube-api-access-bw574" (OuterVolumeSpecName: "kube-api-access-bw574") pod "2217f545-ab39-4f75-9f77-1fdd62d281d8" (UID: "2217f545-ab39-4f75-9f77-1fdd62d281d8"). InnerVolumeSpecName "kube-api-access-bw574". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:45:48.247258 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.247237 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-dshm" (OuterVolumeSpecName: "dshm") pod "2217f545-ab39-4f75-9f77-1fdd62d281d8" (UID: "2217f545-ab39-4f75-9f77-1fdd62d281d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:45:48.247391 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.247237 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2217f545-ab39-4f75-9f77-1fdd62d281d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2217f545-ab39-4f75-9f77-1fdd62d281d8" (UID: "2217f545-ab39-4f75-9f77-1fdd62d281d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:45:48.346522 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.346492 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:48.346522 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.346516 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:48.346522 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.346524 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:48.346706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.346535 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2217f545-ab39-4f75-9f77-1fdd62d281d8-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:48.346706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.346543 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw574\" (UniqueName: \"kubernetes.io/projected/2217f545-ab39-4f75-9f77-1fdd62d281d8-kube-api-access-bw574\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:48.346706 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.346551 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2217f545-ab39-4f75-9f77-1fdd62d281d8-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:45:48.646301 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.646267 2553 generic.go:356] "Generic (PLEG): container finished" podID="2217f545-ab39-4f75-9f77-1fdd62d281d8" containerID="561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610" exitCode=0 Jul 15 21:45:48.646481 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.646314 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" Jul 15 21:45:48.646481 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.646350 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" event={"ID":"2217f545-ab39-4f75-9f77-1fdd62d281d8","Type":"ContainerDied","Data":"561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610"} Jul 15 21:45:48.646481 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.646396 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb" event={"ID":"2217f545-ab39-4f75-9f77-1fdd62d281d8","Type":"ContainerDied","Data":"bbf0597b854a4fed00400c0c42cca002c3dfc3fad139affea350931c390d617d"} Jul 15 21:45:48.646481 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.646418 2553 scope.go:117] "RemoveContainer" containerID="561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610" Jul 15 21:45:48.655109 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.655084 2553 scope.go:117] "RemoveContainer" containerID="561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610" Jul 15 21:45:48.655359 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:45:48.655342 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610\": container with ID starting with 561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610 not found: ID does not exist" containerID="561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610" Jul 15 21:45:48.655406 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.655366 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610"} err="failed to get container status \"561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610\": rpc error: code = NotFound desc = could not find container \"561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610\": container with ID starting with 561d3e13f0573a86b16c5f008b1a1962bc329610a7f42a98b9c06de604302610 not found: ID does not exist" Jul 15 21:45:48.666543 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.666521 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb"] Jul 15 21:45:48.670651 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:48.670630 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/auth-test-model-700cbf-kserve-848fd4cd4c-v4vfb"] Jul 15 21:45:49.143919 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:49.143887 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2217f545-ab39-4f75-9f77-1fdd62d281d8" path="/var/lib/kubelet/pods/2217f545-ab39-4f75-9f77-1fdd62d281d8/volumes" Jul 15 21:45:49.144359 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:49.144344 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed8d1d8-cc09-486f-ae44-be90e46cad6d" path="/var/lib/kubelet/pods/aed8d1d8-cc09-486f-ae44-be90e46cad6d/volumes" Jul 15 21:45:50.640743 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:50.640705 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:50.640743 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:50.640752 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:50.652657 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:50.652627 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:50.674339 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:50.674311 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:45:54.395407 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:45:54.395377 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:46:12.227871 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.227840 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f"] Jul 15 21:46:12.228262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.228157 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed8d1d8-cc09-486f-ae44-be90e46cad6d" containerName="main" Jul 15 21:46:12.228262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.228168 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed8d1d8-cc09-486f-ae44-be90e46cad6d" containerName="main" Jul 15 21:46:12.228262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.228182 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2217f545-ab39-4f75-9f77-1fdd62d281d8" containerName="main" Jul 15 21:46:12.228262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.228188 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="2217f545-ab39-4f75-9f77-1fdd62d281d8" containerName="main" Jul 15 21:46:12.228385 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.228274 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed8d1d8-cc09-486f-ae44-be90e46cad6d" containerName="main" Jul 15 21:46:12.228385 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.228283 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="2217f545-ab39-4f75-9f77-1fdd62d281d8" containerName="main" Jul 15 21:46:12.232526 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.232508 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.234989 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.234964 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"openshift-service-ca.crt\"" Jul 15 21:46:12.235896 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.235876 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"kube-root-ca.crt\"" Jul 15 21:46:12.236016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.235926 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"default-dockercfg-8smwz\"" Jul 15 21:46:12.236016 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.235967 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"sub-test-model-700cbf-kserve-self-signed-certs\"" Jul 15 21:46:12.240943 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.240923 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f"] Jul 15 21:46:12.257860 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.257815 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:46:12.267316 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.267296 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xx5v\" (UniqueName: \"kubernetes.io/projected/ca1c390e-d472-4696-91e0-1164713d3af1-kube-api-access-7xx5v\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.267406 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.267350 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1c390e-d472-4696-91e0-1164713d3af1-tls-certs\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.267406 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.267393 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-home\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.267479 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.267422 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-tmp-dir\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.267513 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.267503 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-dshm\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.267554 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.267541 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-model-cache\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368188 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368156 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-home\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368347 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368196 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-tmp-dir\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368347 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368236 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-dshm\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368347 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368260 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-model-cache\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368347 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368287 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xx5v\" (UniqueName: \"kubernetes.io/projected/ca1c390e-d472-4696-91e0-1164713d3af1-kube-api-access-7xx5v\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368347 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368315 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1c390e-d472-4696-91e0-1164713d3af1-tls-certs\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368589 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368573 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-home\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368640 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368610 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-tmp-dir\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.368677 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.368663 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-model-cache\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.370302 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.370280 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-dshm\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.370615 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.370600 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1c390e-d472-4696-91e0-1164713d3af1-tls-certs\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.376620 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.376597 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xx5v\" (UniqueName: \"kubernetes.io/projected/ca1c390e-d472-4696-91e0-1164713d3af1-kube-api-access-7xx5v\") pod \"sub-test-model-700cbf-kserve-79d5459cff-gwk5f\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.542257 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.542142 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:12.668132 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.668107 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f"] Jul 15 21:46:12.669938 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:46:12.669909 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca1c390e_d472_4696_91e0_1164713d3af1.slice/crio-c61e31e9cf45e60f06cbece3980836d26a00d8aba08c6f2b517cb5d92362e3cc WatchSource:0}: Error finding container c61e31e9cf45e60f06cbece3980836d26a00d8aba08c6f2b517cb5d92362e3cc: Status 404 returned error can't find the container with id c61e31e9cf45e60f06cbece3980836d26a00d8aba08c6f2b517cb5d92362e3cc Jul 15 21:46:12.757630 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.757594 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" event={"ID":"ca1c390e-d472-4696-91e0-1164713d3af1","Type":"ContainerStarted","Data":"e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2"} Jul 15 21:46:12.757804 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.757641 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" event={"ID":"ca1c390e-d472-4696-91e0-1164713d3af1","Type":"ContainerStarted","Data":"c61e31e9cf45e60f06cbece3980836d26a00d8aba08c6f2b517cb5d92362e3cc"} Jul 15 21:46:12.779378 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:12.779318 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" podStartSLOduration=0.779301881 podStartE2EDuration="779.301881ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:46:12.778533868 +0000 UTC m=+2190.905197847" watchObservedRunningTime="2026-07-15 21:46:12.779301881 +0000 UTC m=+2190.905965860" Jul 15 21:46:16.993702 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:16.993670 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:46:22.542287 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:22.542257 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:22.542714 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:22.542299 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:22.554581 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:22.554556 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:22.810454 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:22.810428 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:46:33.299059 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:33.299026 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:46:49.088383 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:49.088353 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp"] Jul 15 21:46:49.108642 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:49.108608 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:46:49.927607 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:49.927575 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f"] Jul 15 21:46:49.947685 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:49.947657 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:46:55.470340 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:55.470301 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:46:56.767398 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:46:56.767362 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:03.714092 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.714060 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp"] Jul 15 21:47:03.717577 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.717555 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.720299 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.720280 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c664d0\"/\"rate-test-model-c664d0-kserve-self-signed-certs\"" Jul 15 21:47:03.728861 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.728838 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp"] Jul 15 21:47:03.753105 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.753083 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:03.807446 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.807415 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-home\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.807557 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.807448 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6sd\" (UniqueName: \"kubernetes.io/projected/9ad5bb45-1bae-43e5-88ef-732378b12737-kube-api-access-kz6sd\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.807557 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.807477 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-tmp-dir\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.807557 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.807541 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-model-cache\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.807661 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.807570 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-dshm\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.807661 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.807586 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad5bb45-1bae-43e5-88ef-732378b12737-tls-certs\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.908821 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.908782 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-home\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.908950 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.908830 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6sd\" (UniqueName: \"kubernetes.io/projected/9ad5bb45-1bae-43e5-88ef-732378b12737-kube-api-access-kz6sd\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.908950 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.908854 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-tmp-dir\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.908950 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.908879 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-model-cache\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.908950 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.908903 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-dshm\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.909141 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.909031 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad5bb45-1bae-43e5-88ef-732378b12737-tls-certs\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.909141 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.909097 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-home\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.909218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.909180 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-tmp-dir\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.909256 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.909227 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-model-cache\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.911190 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.911172 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-dshm\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.911374 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.911357 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad5bb45-1bae-43e5-88ef-732378b12737-tls-certs\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:03.917476 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:03.917454 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6sd\" (UniqueName: \"kubernetes.io/projected/9ad5bb45-1bae-43e5-88ef-732378b12737-kube-api-access-kz6sd\") pod \"rate-test-model-c664d0-kserve-54ccb9754c-27xfp\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:04.026564 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.026487 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:04.107544 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.107478 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" podUID="c0983b6e-fd95-42a8-a27d-b80a1669beb4" containerName="main" containerID="cri-o://9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82" gracePeriod=45 Jul 15 21:47:04.152163 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.152137 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp"] Jul 15 21:47:04.180739 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:47:04.180713 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ad5bb45_1bae_43e5_88ef_732378b12737.slice/crio-90852be8aa3213f00693037cfd7b8e9a41e90207c12fedd102271758b325dbda WatchSource:0}: Error finding container 90852be8aa3213f00693037cfd7b8e9a41e90207c12fedd102271758b325dbda: Status 404 returned error can't find the container with id 90852be8aa3213f00693037cfd7b8e9a41e90207c12fedd102271758b325dbda Jul 15 21:47:04.345434 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.345412 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:47:04.414050 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414019 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-dshm\") pod \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " Jul 15 21:47:04.414194 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414059 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-model-cache\") pod \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " Jul 15 21:47:04.414194 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414085 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tmp-dir\") pod \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " Jul 15 21:47:04.414194 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414129 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzst7\" (UniqueName: \"kubernetes.io/projected/c0983b6e-fd95-42a8-a27d-b80a1669beb4-kube-api-access-tzst7\") pod \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " Jul 15 21:47:04.414194 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414185 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-home\") pod \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " Jul 15 21:47:04.414362 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414279 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0983b6e-fd95-42a8-a27d-b80a1669beb4" (UID: "c0983b6e-fd95-42a8-a27d-b80a1669beb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:04.414362 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414312 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tls-certs\") pod \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\" (UID: \"c0983b6e-fd95-42a8-a27d-b80a1669beb4\") " Jul 15 21:47:04.414362 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414339 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c0983b6e-fd95-42a8-a27d-b80a1669beb4" (UID: "c0983b6e-fd95-42a8-a27d-b80a1669beb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:04.414514 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414412 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-home" (OuterVolumeSpecName: "home") pod "c0983b6e-fd95-42a8-a27d-b80a1669beb4" (UID: "c0983b6e-fd95-42a8-a27d-b80a1669beb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:04.414672 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414651 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:04.414732 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414677 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:04.414732 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.414691 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:04.416098 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.416077 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-dshm" (OuterVolumeSpecName: "dshm") pod "c0983b6e-fd95-42a8-a27d-b80a1669beb4" (UID: "c0983b6e-fd95-42a8-a27d-b80a1669beb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:04.416162 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.416136 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c0983b6e-fd95-42a8-a27d-b80a1669beb4" (UID: "c0983b6e-fd95-42a8-a27d-b80a1669beb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:47:04.416162 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.416142 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0983b6e-fd95-42a8-a27d-b80a1669beb4-kube-api-access-tzst7" (OuterVolumeSpecName: "kube-api-access-tzst7") pod "c0983b6e-fd95-42a8-a27d-b80a1669beb4" (UID: "c0983b6e-fd95-42a8-a27d-b80a1669beb4"). InnerVolumeSpecName "kube-api-access-tzst7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:47:04.515355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.515324 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0983b6e-fd95-42a8-a27d-b80a1669beb4-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:04.515355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.515350 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0983b6e-fd95-42a8-a27d-b80a1669beb4-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:04.515355 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.515359 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzst7\" (UniqueName: \"kubernetes.io/projected/c0983b6e-fd95-42a8-a27d-b80a1669beb4-kube-api-access-tzst7\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:04.946413 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.946376 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" podUID="ca1c390e-d472-4696-91e0-1164713d3af1" containerName="main" containerID="cri-o://e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2" gracePeriod=45 Jul 15 21:47:04.999087 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.999058 2553 generic.go:356] "Generic (PLEG): container finished" podID="c0983b6e-fd95-42a8-a27d-b80a1669beb4" containerID="9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82" exitCode=0 Jul 15 21:47:04.999259 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.999106 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" Jul 15 21:47:04.999259 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.999145 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" event={"ID":"c0983b6e-fd95-42a8-a27d-b80a1669beb4","Type":"ContainerDied","Data":"9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82"} Jul 15 21:47:04.999259 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.999187 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp" event={"ID":"c0983b6e-fd95-42a8-a27d-b80a1669beb4","Type":"ContainerDied","Data":"dba227260bfced2e2cfb786e7b6b0dcba97e31b939f0a36adca2f6728da2cdbd"} Jul 15 21:47:04.999259 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:04.999209 2553 scope.go:117] "RemoveContainer" containerID="9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82" Jul 15 21:47:05.000821 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.000797 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" event={"ID":"9ad5bb45-1bae-43e5-88ef-732378b12737","Type":"ContainerStarted","Data":"e0cd6b9f2aa2589a45a942f012d68b07130680e711614a7856ae641965c71b4b"} Jul 15 21:47:05.000934 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.000831 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" event={"ID":"9ad5bb45-1bae-43e5-88ef-732378b12737","Type":"ContainerStarted","Data":"90852be8aa3213f00693037cfd7b8e9a41e90207c12fedd102271758b325dbda"} Jul 15 21:47:05.020189 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.020149 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" podStartSLOduration=2.020135883 podStartE2EDuration="2.020135883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:47:05.01882829 +0000 UTC m=+2243.145492272" watchObservedRunningTime="2026-07-15 21:47:05.020135883 +0000 UTC m=+2243.146799871" Jul 15 21:47:05.033190 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.033169 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp"] Jul 15 21:47:05.037548 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.037528 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/sub-test-model-c664d0-kserve-68b45d686f-xcmlp"] Jul 15 21:47:05.056703 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.056647 2553 scope.go:117] "RemoveContainer" containerID="9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82" Jul 15 21:47:05.057028 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:47:05.057000 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82\": container with ID starting with 9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82 not found: ID does not exist" containerID="9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82" Jul 15 21:47:05.057110 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.057045 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82"} err="failed to get container status \"9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82\": rpc error: code = NotFound desc = could not find container \"9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82\": container with ID starting with 9870965eb8ab91a0c53a1c94706a7c404e8f12522482d479e41c6edef975de82 not found: ID does not exist" Jul 15 21:47:05.148433 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.148377 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0983b6e-fd95-42a8-a27d-b80a1669beb4" path="/var/lib/kubelet/pods/c0983b6e-fd95-42a8-a27d-b80a1669beb4/volumes" Jul 15 21:47:05.201297 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.201273 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:47:05.324173 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324130 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-tmp-dir\") pod \"ca1c390e-d472-4696-91e0-1164713d3af1\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " Jul 15 21:47:05.324173 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324180 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1c390e-d472-4696-91e0-1164713d3af1-tls-certs\") pod \"ca1c390e-d472-4696-91e0-1164713d3af1\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " Jul 15 21:47:05.324421 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324209 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-home\") pod \"ca1c390e-d472-4696-91e0-1164713d3af1\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " Jul 15 21:47:05.324421 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324230 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-model-cache\") pod \"ca1c390e-d472-4696-91e0-1164713d3af1\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " Jul 15 21:47:05.324421 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324357 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-dshm\") pod \"ca1c390e-d472-4696-91e0-1164713d3af1\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " Jul 15 21:47:05.324569 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324445 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xx5v\" (UniqueName: \"kubernetes.io/projected/ca1c390e-d472-4696-91e0-1164713d3af1-kube-api-access-7xx5v\") pod \"ca1c390e-d472-4696-91e0-1164713d3af1\" (UID: \"ca1c390e-d472-4696-91e0-1164713d3af1\") " Jul 15 21:47:05.324569 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324513 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-home" (OuterVolumeSpecName: "home") pod "ca1c390e-d472-4696-91e0-1164713d3af1" (UID: "ca1c390e-d472-4696-91e0-1164713d3af1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:05.324569 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324519 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca1c390e-d472-4696-91e0-1164713d3af1" (UID: "ca1c390e-d472-4696-91e0-1164713d3af1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:05.324569 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324535 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca1c390e-d472-4696-91e0-1164713d3af1" (UID: "ca1c390e-d472-4696-91e0-1164713d3af1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:05.324786 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324743 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:05.324853 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324792 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:05.324853 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.324806 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:05.326419 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.326391 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-dshm" (OuterVolumeSpecName: "dshm") pod "ca1c390e-d472-4696-91e0-1164713d3af1" (UID: "ca1c390e-d472-4696-91e0-1164713d3af1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:47:05.326538 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.326429 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1c390e-d472-4696-91e0-1164713d3af1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca1c390e-d472-4696-91e0-1164713d3af1" (UID: "ca1c390e-d472-4696-91e0-1164713d3af1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:47:05.326538 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.326474 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1c390e-d472-4696-91e0-1164713d3af1-kube-api-access-7xx5v" (OuterVolumeSpecName: "kube-api-access-7xx5v") pod "ca1c390e-d472-4696-91e0-1164713d3af1" (UID: "ca1c390e-d472-4696-91e0-1164713d3af1"). InnerVolumeSpecName "kube-api-access-7xx5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:47:05.426089 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.426049 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xx5v\" (UniqueName: \"kubernetes.io/projected/ca1c390e-d472-4696-91e0-1164713d3af1-kube-api-access-7xx5v\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:05.426089 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.426083 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca1c390e-d472-4696-91e0-1164713d3af1-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:05.426089 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:05.426096 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca1c390e-d472-4696-91e0-1164713d3af1-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:47:06.006461 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.006424 2553 generic.go:356] "Generic (PLEG): container finished" podID="ca1c390e-d472-4696-91e0-1164713d3af1" containerID="e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2" exitCode=0 Jul 15 21:47:06.006921 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.006463 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" Jul 15 21:47:06.006921 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.006517 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" event={"ID":"ca1c390e-d472-4696-91e0-1164713d3af1","Type":"ContainerDied","Data":"e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2"} Jul 15 21:47:06.006921 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.006553 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f" event={"ID":"ca1c390e-d472-4696-91e0-1164713d3af1","Type":"ContainerDied","Data":"c61e31e9cf45e60f06cbece3980836d26a00d8aba08c6f2b517cb5d92362e3cc"} Jul 15 21:47:06.006921 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.006569 2553 scope.go:117] "RemoveContainer" containerID="e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2" Jul 15 21:47:06.014903 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.014877 2553 scope.go:117] "RemoveContainer" containerID="e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2" Jul 15 21:47:06.015195 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:47:06.015177 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2\": container with ID starting with e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2 not found: ID does not exist" containerID="e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2" Jul 15 21:47:06.015275 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.015201 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2"} err="failed to get container status \"e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2\": rpc error: code = NotFound desc = could not find container \"e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2\": container with ID starting with e6dd8856d4c05373b64eeb98102f2f5f0cd8b64db5ddea056df50b7c067105f2 not found: ID does not exist" Jul 15 21:47:06.026438 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.026329 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f"] Jul 15 21:47:06.030866 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.030846 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/sub-test-model-700cbf-kserve-79d5459cff-gwk5f"] Jul 15 21:47:06.697154 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:06.697120 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:47:07.145822 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:07.145783 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1c390e-d472-4696-91e0-1164713d3af1" path="/var/lib/kubelet/pods/ca1c390e-d472-4696-91e0-1164713d3af1/volumes" Jul 15 21:47:14.027122 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:14.027077 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:14.027566 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:14.027144 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:14.039289 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:14.039256 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:14.053840 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:14.053816 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:47:41.723790 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:41.723721 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:41.823469 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:41.823436 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:42.030083 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.030007 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j"] Jul 15 21:47:42.030375 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.030363 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0983b6e-fd95-42a8-a27d-b80a1669beb4" containerName="main" Jul 15 21:47:42.030455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.030378 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0983b6e-fd95-42a8-a27d-b80a1669beb4" containerName="main" Jul 15 21:47:42.030455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.030398 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1c390e-d472-4696-91e0-1164713d3af1" containerName="main" Jul 15 21:47:42.030455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.030404 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1c390e-d472-4696-91e0-1164713d3af1" containerName="main" Jul 15 21:47:42.030544 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.030459 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0983b6e-fd95-42a8-a27d-b80a1669beb4" containerName="main" Jul 15 21:47:42.030544 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.030470 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1c390e-d472-4696-91e0-1164713d3af1" containerName="main" Jul 15 21:47:42.033498 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.033481 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.037045 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.037024 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"kube-root-ca.crt\"" Jul 15 21:47:42.037168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.037065 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"default-dockercfg-8smwz\"" Jul 15 21:47:42.037168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.037068 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"openshift-service-ca.crt\"" Jul 15 21:47:42.037168 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.037107 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"rate-test-model-700cbf-kserve-self-signed-certs\"" Jul 15 21:47:42.044002 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.043980 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j"] Jul 15 21:47:42.140138 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.140105 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-home\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.140138 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.140143 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3c352-35f3-4bdd-8b6e-175872527366-tls-certs\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.140338 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.140213 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-tmp-dir\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.140338 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.140278 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-model-cache\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.140338 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.140297 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkklp\" (UniqueName: \"kubernetes.io/projected/73c3c352-35f3-4bdd-8b6e-175872527366-kube-api-access-nkklp\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.140452 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.140361 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-dshm\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241115 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241077 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-dshm\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241276 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241122 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-home\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241276 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241150 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3c352-35f3-4bdd-8b6e-175872527366-tls-certs\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241276 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241201 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-tmp-dir\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241411 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241272 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-model-cache\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241411 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241299 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkklp\" (UniqueName: \"kubernetes.io/projected/73c3c352-35f3-4bdd-8b6e-175872527366-kube-api-access-nkklp\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241482 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241453 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-home\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241868 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241847 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-tmp-dir\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.241992 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.241941 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-model-cache\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.243557 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.243537 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-dshm\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.243619 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.243600 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3c352-35f3-4bdd-8b6e-175872527366-tls-certs\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.252073 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.252052 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkklp\" (UniqueName: \"kubernetes.io/projected/73c3c352-35f3-4bdd-8b6e-175872527366-kube-api-access-nkklp\") pod \"rate-test-model-700cbf-kserve-655c6b5459-prk8j\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.280345 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.280266 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:42.344329 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.344307 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-700cbf\"/\"default-dockercfg-8smwz\"" Jul 15 21:47:42.352390 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.352374 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:42.477528 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:42.477503 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j"] Jul 15 21:47:42.479286 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:47:42.479260 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73c3c352_35f3_4bdd_8b6e_175872527366.slice/crio-7b545b9fa024cd4bb5e043bef218f2c1d15ba5718a3c9c141077bed5694bebcb WatchSource:0}: Error finding container 7b545b9fa024cd4bb5e043bef218f2c1d15ba5718a3c9c141077bed5694bebcb: Status 404 returned error can't find the container with id 7b545b9fa024cd4bb5e043bef218f2c1d15ba5718a3c9c141077bed5694bebcb Jul 15 21:47:43.168280 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:43.168254 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" event={"ID":"73c3c352-35f3-4bdd-8b6e-175872527366","Type":"ContainerStarted","Data":"1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7"} Jul 15 21:47:43.168579 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:43.168287 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" event={"ID":"73c3c352-35f3-4bdd-8b6e-175872527366","Type":"ContainerStarted","Data":"7b545b9fa024cd4bb5e043bef218f2c1d15ba5718a3c9c141077bed5694bebcb"} Jul 15 21:47:43.189350 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:43.189308 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" podStartSLOduration=1.189293223 podStartE2EDuration="1.189293223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:47:43.187462899 +0000 UTC m=+2281.314126881" watchObservedRunningTime="2026-07-15 21:47:43.189293223 +0000 UTC m=+2281.315957203" Jul 15 21:47:46.797636 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:46.797603 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:47:50.522218 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:50.522182 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:50.626619 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:50.626585 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:52.352813 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:52.352774 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:52.353225 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:52.352826 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:52.364984 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:52.364958 2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:47:53.221298 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:47:53.221269 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:48:03.093716 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:03.093685 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:48:27.708636 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:27.708602 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:48:32.197751 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:32.197719 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp"] Jul 15 21:48:32.240071 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:32.240031 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:33.036436 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:33.036386 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j"] Jul 15 21:48:33.061943 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:33.061907 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:44.795349 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:44.795267 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:48:47.218149 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.218112 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" podUID="9ad5bb45-1bae-43e5-88ef-732378b12737" containerName="main" containerID="cri-o://e0cd6b9f2aa2589a45a942f012d68b07130680e711614a7856ae641965c71b4b" gracePeriod=45 Jul 15 21:48:47.449361 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.449333 2553 generic.go:356] "Generic (PLEG): container finished" podID="9ad5bb45-1bae-43e5-88ef-732378b12737" containerID="e0cd6b9f2aa2589a45a942f012d68b07130680e711614a7856ae641965c71b4b" exitCode=0 Jul 15 21:48:47.449493 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.449403 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" event={"ID":"9ad5bb45-1bae-43e5-88ef-732378b12737","Type":"ContainerDied","Data":"e0cd6b9f2aa2589a45a942f012d68b07130680e711614a7856ae641965c71b4b"} Jul 15 21:48:47.461521 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.461498 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:48:47.527501 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527428 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-model-cache\") pod \"9ad5bb45-1bae-43e5-88ef-732378b12737\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " Jul 15 21:48:47.527501 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527472 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz6sd\" (UniqueName: \"kubernetes.io/projected/9ad5bb45-1bae-43e5-88ef-732378b12737-kube-api-access-kz6sd\") pod \"9ad5bb45-1bae-43e5-88ef-732378b12737\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " Jul 15 21:48:47.527501 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527499 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-dshm\") pod \"9ad5bb45-1bae-43e5-88ef-732378b12737\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " Jul 15 21:48:47.527801 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527594 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad5bb45-1bae-43e5-88ef-732378b12737-tls-certs\") pod \"9ad5bb45-1bae-43e5-88ef-732378b12737\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " Jul 15 21:48:47.527801 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527633 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-home\") pod \"9ad5bb45-1bae-43e5-88ef-732378b12737\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " Jul 15 21:48:47.527801 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527696 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-tmp-dir\") pod \"9ad5bb45-1bae-43e5-88ef-732378b12737\" (UID: \"9ad5bb45-1bae-43e5-88ef-732378b12737\") " Jul 15 21:48:47.527801 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527707 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-model-cache" (OuterVolumeSpecName: "model-cache") pod "9ad5bb45-1bae-43e5-88ef-732378b12737" (UID: "9ad5bb45-1bae-43e5-88ef-732378b12737"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:47.528011 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527920 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-home" (OuterVolumeSpecName: "home") pod "9ad5bb45-1bae-43e5-88ef-732378b12737" (UID: "9ad5bb45-1bae-43e5-88ef-732378b12737"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:47.528011 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.527935 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9ad5bb45-1bae-43e5-88ef-732378b12737" (UID: "9ad5bb45-1bae-43e5-88ef-732378b12737"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:47.528112 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.528013 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:47.528112 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.528031 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:47.528112 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.528044 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:47.529579 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.529550 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad5bb45-1bae-43e5-88ef-732378b12737-kube-api-access-kz6sd" (OuterVolumeSpecName: "kube-api-access-kz6sd") pod "9ad5bb45-1bae-43e5-88ef-732378b12737" (UID: "9ad5bb45-1bae-43e5-88ef-732378b12737"). InnerVolumeSpecName "kube-api-access-kz6sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:47.529990 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.529969 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ad5bb45-1bae-43e5-88ef-732378b12737-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ad5bb45-1bae-43e5-88ef-732378b12737" (UID: "9ad5bb45-1bae-43e5-88ef-732378b12737"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:48:47.529990 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.529980 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-dshm" (OuterVolumeSpecName: "dshm") pod "9ad5bb45-1bae-43e5-88ef-732378b12737" (UID: "9ad5bb45-1bae-43e5-88ef-732378b12737"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:47.628805 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.628743 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ad5bb45-1bae-43e5-88ef-732378b12737-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:47.628805 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.628801 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz6sd\" (UniqueName: \"kubernetes.io/projected/9ad5bb45-1bae-43e5-88ef-732378b12737-kube-api-access-kz6sd\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:47.628805 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:47.628812 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ad5bb45-1bae-43e5-88ef-732378b12737-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:48.056610 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.056564 2553 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" podUID="73c3c352-35f3-4bdd-8b6e-175872527366" containerName="main" containerID="cri-o://1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7" gracePeriod=45 Jul 15 21:48:48.294975 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.294949 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:48:48.435753 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.435648 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3c352-35f3-4bdd-8b6e-175872527366-tls-certs\") pod \"73c3c352-35f3-4bdd-8b6e-175872527366\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " Jul 15 21:48:48.435753 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.435737 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkklp\" (UniqueName: \"kubernetes.io/projected/73c3c352-35f3-4bdd-8b6e-175872527366-kube-api-access-nkklp\") pod \"73c3c352-35f3-4bdd-8b6e-175872527366\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " Jul 15 21:48:48.436010 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.435793 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-dshm\") pod \"73c3c352-35f3-4bdd-8b6e-175872527366\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " Jul 15 21:48:48.436010 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.435855 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-tmp-dir\") pod \"73c3c352-35f3-4bdd-8b6e-175872527366\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " Jul 15 21:48:48.436010 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.435881 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-home\") pod \"73c3c352-35f3-4bdd-8b6e-175872527366\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " Jul 15 21:48:48.436010 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.435901 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-model-cache\") pod \"73c3c352-35f3-4bdd-8b6e-175872527366\" (UID: \"73c3c352-35f3-4bdd-8b6e-175872527366\") " Jul 15 21:48:48.436182 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.436095 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "73c3c352-35f3-4bdd-8b6e-175872527366" (UID: "73c3c352-35f3-4bdd-8b6e-175872527366"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:48.436182 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.436111 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-home" (OuterVolumeSpecName: "home") pod "73c3c352-35f3-4bdd-8b6e-175872527366" (UID: "73c3c352-35f3-4bdd-8b6e-175872527366"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:48.436262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.436194 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-model-cache" (OuterVolumeSpecName: "model-cache") pod "73c3c352-35f3-4bdd-8b6e-175872527366" (UID: "73c3c352-35f3-4bdd-8b6e-175872527366"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:48.436262 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.436244 2553 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-tmp-dir\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:48.436329 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.436264 2553 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-home\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:48.437865 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.437846 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-dshm" (OuterVolumeSpecName: "dshm") pod "73c3c352-35f3-4bdd-8b6e-175872527366" (UID: "73c3c352-35f3-4bdd-8b6e-175872527366"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:48:48.437957 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.437924 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c3c352-35f3-4bdd-8b6e-175872527366-kube-api-access-nkklp" (OuterVolumeSpecName: "kube-api-access-nkklp") pod "73c3c352-35f3-4bdd-8b6e-175872527366" (UID: "73c3c352-35f3-4bdd-8b6e-175872527366"). InnerVolumeSpecName "kube-api-access-nkklp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:48.438013 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.437990 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c3c352-35f3-4bdd-8b6e-175872527366-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73c3c352-35f3-4bdd-8b6e-175872527366" (UID: "73c3c352-35f3-4bdd-8b6e-175872527366"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:48:48.454932 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.454911 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" Jul 15 21:48:48.455040 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.454953 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp" event={"ID":"9ad5bb45-1bae-43e5-88ef-732378b12737","Type":"ContainerDied","Data":"90852be8aa3213f00693037cfd7b8e9a41e90207c12fedd102271758b325dbda"} Jul 15 21:48:48.455040 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.454999 2553 scope.go:117] "RemoveContainer" containerID="e0cd6b9f2aa2589a45a942f012d68b07130680e711614a7856ae641965c71b4b" Jul 15 21:48:48.456213 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.456185 2553 generic.go:356] "Generic (PLEG): container finished" podID="73c3c352-35f3-4bdd-8b6e-175872527366" containerID="1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7" exitCode=0 Jul 15 21:48:48.456316 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.456231 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" Jul 15 21:48:48.456316 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.456217 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" event={"ID":"73c3c352-35f3-4bdd-8b6e-175872527366","Type":"ContainerDied","Data":"1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7"} Jul 15 21:48:48.456428 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.456327 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j" event={"ID":"73c3c352-35f3-4bdd-8b6e-175872527366","Type":"ContainerDied","Data":"7b545b9fa024cd4bb5e043bef218f2c1d15ba5718a3c9c141077bed5694bebcb"} Jul 15 21:48:48.463862 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.463838 2553 scope.go:117] "RemoveContainer" containerID="1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7" Jul 15 21:48:48.470856 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.470841 2553 scope.go:117] "RemoveContainer" containerID="1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7" Jul 15 21:48:48.471117 ip-10-0-136-220 kubenswrapper[2553]: E0715 21:48:48.471094 2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7\": container with ID starting with 1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7 not found: ID does not exist" containerID="1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7" Jul 15 21:48:48.471183 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.471128 2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7"} err="failed to get container status \"1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7\": rpc error: code = NotFound desc = could not find container \"1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7\": container with ID starting with 1b2a522b7bd4a5d35add25241a5f52f05a69b59a8e254dd6055c26b49fb9bef7 not found: ID does not exist" Jul 15 21:48:48.475963 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.475944 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp"] Jul 15 21:48:48.479676 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.479657 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c664d0/rate-test-model-c664d0-kserve-54ccb9754c-27xfp"] Jul 15 21:48:48.489584 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.489562 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j"] Jul 15 21:48:48.491622 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.491604 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-700cbf/rate-test-model-700cbf-kserve-655c6b5459-prk8j"] Jul 15 21:48:48.537664 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.537637 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkklp\" (UniqueName: \"kubernetes.io/projected/73c3c352-35f3-4bdd-8b6e-175872527366-kube-api-access-nkklp\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:48.537664 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.537663 2553 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-dshm\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:48.537831 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.537678 2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3c352-35f3-4bdd-8b6e-175872527366-model-cache\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:48.537831 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:48.537692 2553 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3c352-35f3-4bdd-8b6e-175872527366-tls-certs\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:49.149296 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.149239 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c3c352-35f3-4bdd-8b6e-175872527366" path="/var/lib/kubelet/pods/73c3c352-35f3-4bdd-8b6e-175872527366/volumes" Jul 15 21:48:49.149905 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.149883 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ad5bb45-1bae-43e5-88ef-732378b12737" path="/var/lib/kubelet/pods/9ad5bb45-1bae-43e5-88ef-732378b12737/volumes" Jul 15 21:48:49.251691 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.251659 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7624-140173648990544"] Jul 15 21:48:49.251993 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.251979 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c3c352-35f3-4bdd-8b6e-175872527366" containerName="main" Jul 15 21:48:49.252038 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.251994 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c3c352-35f3-4bdd-8b6e-175872527366" containerName="main" Jul 15 21:48:49.252038 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.252012 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ad5bb45-1bae-43e5-88ef-732378b12737" containerName="main" Jul 15 21:48:49.252038 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.252017 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ad5bb45-1bae-43e5-88ef-732378b12737" containerName="main" Jul 15 21:48:49.252123 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.252069 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c3c352-35f3-4bdd-8b6e-175872527366" containerName="main" Jul 15 21:48:49.252123 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.252077 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ad5bb45-1bae-43e5-88ef-732378b12737" containerName="main" Jul 15 21:48:49.256157 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.256142 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-140173648990544" Jul 15 21:48:49.258346 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.258327 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qt2bh\"" Jul 15 21:48:49.261090 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.261070 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-140173648990544"] Jul 15 21:48:49.348944 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.348899 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hm7\" (UniqueName: \"kubernetes.io/projected/7a4be991-cd79-4018-b541-f2387358dd05-kube-api-access-66hm7\") pod \"test-curl-7624-140173648990544\" (UID: \"7a4be991-cd79-4018-b541-f2387358dd05\") " pod="opendatahub/test-curl-7624-140173648990544" Jul 15 21:48:49.450359 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.450274 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hm7\" (UniqueName: \"kubernetes.io/projected/7a4be991-cd79-4018-b541-f2387358dd05-kube-api-access-66hm7\") pod \"test-curl-7624-140173648990544\" (UID: \"7a4be991-cd79-4018-b541-f2387358dd05\") " pod="opendatahub/test-curl-7624-140173648990544" Jul 15 21:48:49.458458 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.458424 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hm7\" (UniqueName: \"kubernetes.io/projected/7a4be991-cd79-4018-b541-f2387358dd05-kube-api-access-66hm7\") pod \"test-curl-7624-140173648990544\" (UID: \"7a4be991-cd79-4018-b541-f2387358dd05\") " pod="opendatahub/test-curl-7624-140173648990544" Jul 15 21:48:49.565524 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.565486 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-140173648990544" Jul 15 21:48:49.689098 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.689074 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7624-140173648990544"] Jul 15 21:48:49.690838 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:48:49.690806 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a4be991_cd79_4018_b541_f2387358dd05.slice/crio-7a7fca235683507ec1f61ee2cb662d98f3e0de702fce2233beaa4fc77e5425dd WatchSource:0}: Error finding container 7a7fca235683507ec1f61ee2cb662d98f3e0de702fce2233beaa4fc77e5425dd: Status 404 returned error can't find the container with id 7a7fca235683507ec1f61ee2cb662d98f3e0de702fce2233beaa4fc77e5425dd Jul 15 21:48:49.692112 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:49.692094 2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:48:50.467412 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:50.467372 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-140173648990544" event={"ID":"7a4be991-cd79-4018-b541-f2387358dd05","Type":"ContainerStarted","Data":"75dcc172adf5156f5f6a5675b8d0ceb28d258b8cc0f0165b8f6dfc51f02ba0ac"} Jul 15 21:48:50.467412 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:50.467417 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-140173648990544" event={"ID":"7a4be991-cd79-4018-b541-f2387358dd05","Type":"ContainerStarted","Data":"7a7fca235683507ec1f61ee2cb662d98f3e0de702fce2233beaa4fc77e5425dd"} Jul 15 21:48:50.483057 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:50.483013 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7624-140173648990544" podStartSLOduration=0.801920387 podStartE2EDuration="1.482998082s" totalImagesPullingTime="681.077695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:48:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:48:50.482220122 +0000 UTC m=+2348.608884101" watchObservedRunningTime="2026-07-15 21:48:50.482998082 +0000 UTC m=+2348.609662052" Jul 15 21:48:50.586579 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:50.586551 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7624-140173648990544_7a4be991-cd79-4018-b541-f2387358dd05/test-curl-7624-140173648990544/0.log" Jul 15 21:48:51.055075 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:51.055042 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:51.155963 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:51.155930 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:51.472881 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:51.472793 2553 generic.go:356] "Generic (PLEG): container finished" podID="7a4be991-cd79-4018-b541-f2387358dd05" containerID="75dcc172adf5156f5f6a5675b8d0ceb28d258b8cc0f0165b8f6dfc51f02ba0ac" exitCode=0 Jul 15 21:48:51.472881 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:51.472861 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7624-140173648990544" event={"ID":"7a4be991-cd79-4018-b541-f2387358dd05","Type":"ContainerDied","Data":"75dcc172adf5156f5f6a5675b8d0ceb28d258b8cc0f0165b8f6dfc51f02ba0ac"} Jul 15 21:48:52.601812 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:52.601785 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-140173648990544" Jul 15 21:48:52.649027 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:52.648999 2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7624-140173648990544"] Jul 15 21:48:52.653314 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:52.653293 2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7624-140173648990544"] Jul 15 21:48:52.679101 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:52.679079 2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66hm7\" (UniqueName: \"kubernetes.io/projected/7a4be991-cd79-4018-b541-f2387358dd05-kube-api-access-66hm7\") pod \"7a4be991-cd79-4018-b541-f2387358dd05\" (UID: \"7a4be991-cd79-4018-b541-f2387358dd05\") " Jul 15 21:48:52.681133 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:52.681106 2553 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a4be991-cd79-4018-b541-f2387358dd05-kube-api-access-66hm7" (OuterVolumeSpecName: "kube-api-access-66hm7") pod "7a4be991-cd79-4018-b541-f2387358dd05" (UID: "7a4be991-cd79-4018-b541-f2387358dd05"). InnerVolumeSpecName "kube-api-access-66hm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:48:52.779629 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:52.779551 2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66hm7\" (UniqueName: \"kubernetes.io/projected/7a4be991-cd79-4018-b541-f2387358dd05-kube-api-access-66hm7\") on node \"ip-10-0-136-220.ec2.internal\" DevicePath \"\"" Jul 15 21:48:53.142844 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:53.142811 2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a4be991-cd79-4018-b541-f2387358dd05" path="/var/lib/kubelet/pods/7a4be991-cd79-4018-b541-f2387358dd05/volumes" Jul 15 21:48:53.247986 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:53.247949 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:53.482216 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:53.482143 2553 scope.go:117] "RemoveContainer" containerID="75dcc172adf5156f5f6a5675b8d0ceb28d258b8cc0f0165b8f6dfc51f02ba0ac" Jul 15 21:48:53.482216 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:53.482158 2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7624-140173648990544" Jul 15 21:48:59.128988 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:59.128957 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:59.231083 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:59.231014 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:59.422745 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:59.422667 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:59.641362 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:48:59.641327 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:00.114017 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:00.113986 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:02.790704 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:02.790669 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:49:06.060260 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:06.060231 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:06.160543 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:06.160511 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:06.347078 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:06.347001 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:06.567871 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:06.567829 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:13.673040 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:13.673012 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d5d77c45b-fr6pn_5506ba67-aa68-47fb-b001-d5fc9074531b/copy-manifests/0.log" Jul 15 21:49:13.681857 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:13.681825 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d5d77c45b-fr6pn_5506ba67-aa68-47fb-b001-d5fc9074531b/manager/0.log" Jul 15 21:49:14.309993 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:14.309970 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-api-a-599e95-67c64789c8-24brf_b0b19be8-6bbc-4dbb-b8c4-23c05b12ef57/maas-api/0.log" Jul 15 21:49:14.433165 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:14.433133 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-api-b-d3a40f-d8b5c8677-9g8jf_86daee49-c31a-4efd-ba30-978782d82dc3/maas-api/0.log" Jul 15 21:49:14.804650 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:14.804623 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-a5309f-d84f8d4cb-4wsl5_b788e6c1-c6e3-47c8-b277-407731d1ce91/maas-api/0.log" Jul 15 21:49:14.952157 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:14.952111 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-migrate-620eb2f1-5f479cff7c-tl77j_ee551edc-421e-4b64-ac1d-3f12088b5282/maas-api/0.log" Jul 15 21:49:15.076370 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:15.076290 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-mt-c2c3a008-85947d79c7-2r4xq_e9b50ed3-0339-4479-bca1-e59b95e48ea1/maas-api/0.log" Jul 15 21:49:16.087483 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:16.087450 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-kvffp"] Jul 15 21:49:18.700824 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:18.700797 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w6sgf_8164491b-a8ff-4353-8d77-f58a36a6c4ab/kuadrant-console-plugin/0.log" Jul 15 21:49:19.145497 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:19.145471 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-kvffp_85a317db-7cba-4e98-935b-5353a133aaf3/limitador/0.log" Jul 15 21:49:20.094082 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:20.094054 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cbf464cb7-bvmzf_e9085bd7-6531-4cc4-abfa-37981373f289/kube-auth-proxy/0.log" Jul 15 21:49:21.251572 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:21.251545 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zgzzs_2a3832a1-d1c2-40bc-a9b7-85c5ef6aa548/main/0.log" Jul 15 21:49:21.627386 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:21.627353 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xc5vb_bb856868-bee2-435f-8f1b-5f89edc26216/main/0.log" Jul 15 21:49:27.636446 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:27.636416 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:27.730798 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:27.730743 2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" limit=1024 Jul 15 21:49:30.689956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:30.689915 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w5bkr_47faff50-dced-41ee-b2da-a1ccb917bb9f/global-pull-secret-syncer/0.log" Jul 15 21:49:30.740368 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:30.740339 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2wxrz_4cfec505-8d15-44c1-af49-2328d48f3a70/konnectivity-agent/0.log" Jul 15 21:49:30.862090 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:30.862062 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-220.ec2.internal_45606745bd8d4c83782613a3e901378b/haproxy/0.log" Jul 15 21:49:36.625885 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:36.625849 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w6sgf_8164491b-a8ff-4353-8d77-f58a36a6c4ab/kuadrant-console-plugin/0.log" Jul 15 21:49:36.813427 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:36.813401 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-kvffp_85a317db-7cba-4e98-935b-5353a133aaf3/limitador/0.log" Jul 15 21:49:38.908455 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:38.908432 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6f5744bf5d-jnc9x_8cbf6d1c-e8d4-47c1-88c8-ae2dabf2e330/metrics-server/0.log" Jul 15 21:49:39.047733 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.047711 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh8dd_67a279c1-1a0e-41d1-9a25-e076e673d17e/node-exporter/0.log" Jul 15 21:49:39.068602 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.068576 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh8dd_67a279c1-1a0e-41d1-9a25-e076e673d17e/kube-rbac-proxy/0.log" Jul 15 21:49:39.087662 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.087642 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fh8dd_67a279c1-1a0e-41d1-9a25-e076e673d17e/init-textfile/0.log" Jul 15 21:49:39.272155 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.272079 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b4333ce-cb9f-430d-ad4b-c499a3b41f0d/prometheus/0.log" Jul 15 21:49:39.292065 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.292041 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b4333ce-cb9f-430d-ad4b-c499a3b41f0d/config-reloader/0.log" Jul 15 21:49:39.314120 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.314087 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b4333ce-cb9f-430d-ad4b-c499a3b41f0d/thanos-sidecar/0.log" Jul 15 21:49:39.334642 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.334625 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b4333ce-cb9f-430d-ad4b-c499a3b41f0d/kube-rbac-proxy-web/0.log" Jul 15 21:49:39.355919 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.355883 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b4333ce-cb9f-430d-ad4b-c499a3b41f0d/kube-rbac-proxy/0.log" Jul 15 21:49:39.376246 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.376232 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b4333ce-cb9f-430d-ad4b-c499a3b41f0d/kube-rbac-proxy-thanos/0.log" Jul 15 21:49:39.396468 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:39.396454 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0b4333ce-cb9f-430d-ad4b-c499a3b41f0d/init-config-reloader/0.log" Jul 15 21:49:40.854557 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.854529 2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp"] Jul 15 21:49:40.854924 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.854882 2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a4be991-cd79-4018-b541-f2387358dd05" containerName="test-curl-7624-140173648990544" Jul 15 21:49:40.854924 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.854895 2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a4be991-cd79-4018-b541-f2387358dd05" containerName="test-curl-7624-140173648990544" Jul 15 21:49:40.854994 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.854949 2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a4be991-cd79-4018-b541-f2387358dd05" containerName="test-curl-7624-140173648990544" Jul 15 21:49:40.859227 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.859210 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:40.862095 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.862070 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hvsf\"/\"kube-root-ca.crt\"" Jul 15 21:49:40.862213 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.862146 2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hvsf\"/\"default-dockercfg-vhb5k\"" Jul 15 21:49:40.863015 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.862994 2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hvsf\"/\"openshift-service-ca.crt\"" Jul 15 21:49:40.865392 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.865371 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp"] Jul 15 21:49:40.989448 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.989412 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-podres\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:40.989618 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.989460 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-proc\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:40.989618 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.989510 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-sys\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:40.989618 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.989529 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngc2\" (UniqueName: \"kubernetes.io/projected/c66f06fb-950a-435e-96e5-53851d03597e-kube-api-access-gngc2\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:40.989618 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:40.989584 2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-lib-modules\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090295 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090262 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-proc\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090295 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090299 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-sys\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090317 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngc2\" (UniqueName: \"kubernetes.io/projected/c66f06fb-950a-435e-96e5-53851d03597e-kube-api-access-gngc2\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090354 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-lib-modules\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090389 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-proc\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090395 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-sys\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090398 2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-podres\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090509 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090489 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-lib-modules\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.090719 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.090554 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c66f06fb-950a-435e-96e5-53851d03597e-podres\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.098282 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.098253 2553 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngc2\" (UniqueName: \"kubernetes.io/projected/c66f06fb-950a-435e-96e5-53851d03597e-kube-api-access-gngc2\") pod \"perf-node-gather-daemonset-kp4dp\" (UID: \"c66f06fb-950a-435e-96e5-53851d03597e\") " pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.169755 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.169669 2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.294933 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.294878 2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp"] Jul 15 21:49:41.297132 ip-10-0-136-220 kubenswrapper[2553]: W0715 21:49:41.297101 2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc66f06fb_950a_435e_96e5_53851d03597e.slice/crio-f0d634e6c4d8abac0879bbd77c6e589e85a8dea924b6fa3a7d321e743ff7db13 WatchSource:0}: Error finding container f0d634e6c4d8abac0879bbd77c6e589e85a8dea924b6fa3a7d321e743ff7db13: Status 404 returned error can't find the container with id f0d634e6c4d8abac0879bbd77c6e589e85a8dea924b6fa3a7d321e743ff7db13 Jul 15 21:49:41.694971 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.694939 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" event={"ID":"c66f06fb-950a-435e-96e5-53851d03597e","Type":"ContainerStarted","Data":"09467e033797056cf2c268938a1c63208598abbe83ff16cdaea8e09c046fad02"} Jul 15 21:49:41.695132 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.694989 2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" event={"ID":"c66f06fb-950a-435e-96e5-53851d03597e","Type":"ContainerStarted","Data":"f0d634e6c4d8abac0879bbd77c6e589e85a8dea924b6fa3a7d321e743ff7db13"} Jul 15 21:49:41.695132 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.695052 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:41.711432 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:41.711391 2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" podStartSLOduration=1.711379676 podStartE2EDuration="1.711379676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:49:41.709715983 +0000 UTC m=+2399.836380000" watchObservedRunningTime="2026-07-15 21:49:41.711379676 +0000 UTC m=+2399.838043656" Jul 15 21:49:43.102121 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:43.102094 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:49:43.108294 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:43.108276 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zqbh7_7d7cd9da-9939-42a9-9a3e-56129fc6f090/ovn-acl-logging/0.log" Jul 15 21:49:43.717572 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:43.717546 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jmv6d_2cdb38bf-4fe4-4d06-8e86-109d1e447b69/dns/0.log" Jul 15 21:49:43.739071 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:43.739047 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jmv6d_2cdb38bf-4fe4-4d06-8e86-109d1e447b69/kube-rbac-proxy/0.log" Jul 15 21:49:43.880328 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:43.880303 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lqmdr_00459e80-47cb-4df5-a7d0-4a6c6d0a173c/dns-node-resolver/0.log" Jul 15 21:49:44.459842 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:44.459807 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k957b_82ffd961-41ee-44d3-b38f-c74dafaa59ca/node-ca/0.log" Jul 15 21:49:45.573022 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:45.572989 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cbf464cb7-bvmzf_e9085bd7-6531-4cc4-abfa-37981373f289/kube-auth-proxy/0.log" Jul 15 21:49:46.450411 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:46.450385 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m24rn_af8aa079-6a91-4be0-ad4a-65c8de7ccf07/serve-healthcheck-canary/0.log" Jul 15 21:49:46.974350 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:46.974325 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjbs6_013328ec-a9d4-457e-b5d2-6944a2196aa5/kube-rbac-proxy/0.log" Jul 15 21:49:46.994964 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:46.994938 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjbs6_013328ec-a9d4-457e-b5d2-6944a2196aa5/exporter/0.log" Jul 15 21:49:47.015820 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:47.015804 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bjbs6_013328ec-a9d4-457e-b5d2-6944a2196aa5/extractor/0.log" Jul 15 21:49:47.707962 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:47.707933 2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8hvsf/perf-node-gather-daemonset-kp4dp" Jul 15 21:49:50.198452 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:50.198420 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d5d77c45b-fr6pn_5506ba67-aa68-47fb-b001-d5fc9074531b/manager/0.log" Jul 15 21:49:50.222267 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:50.222239 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d5d77c45b-fr6pn_5506ba67-aa68-47fb-b001-d5fc9074531b/copy-manifests/0.log" Jul 15 21:49:51.456089 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:51.456062 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cbf464cb7-bvmzf_e9085bd7-6531-4cc4-abfa-37981373f289/kube-auth-proxy/0.log" Jul 15 21:49:51.538617 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:51.538590 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cd5bf8cdd-whjf2_bb2bfb76-73b4-4801-a9cd-6ca7eba8e7c5/manager/0.log" Jul 15 21:49:53.006656 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:53.006626 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cbf464cb7-bvmzf_e9085bd7-6531-4cc4-abfa-37981373f289/kube-auth-proxy/0.log" Jul 15 21:49:56.467998 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:56.467917 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z6khg_afc5e170-3cf3-4510-9498-101aad6b7c49/migrator/0.log" Jul 15 21:49:56.488738 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:56.488714 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-z6khg_afc5e170-3cf3-4510-9498-101aad6b7c49/graceful-termination/0.log" Jul 15 21:49:58.075956 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.075926 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vchqc_ca0d0d1d-4805-4ad6-9b4a-38b0e621affc/kube-multus-additional-cni-plugins/0.log" Jul 15 21:49:58.095875 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.095852 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vchqc_ca0d0d1d-4805-4ad6-9b4a-38b0e621affc/egress-router-binary-copy/0.log" Jul 15 21:49:58.115650 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.115632 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vchqc_ca0d0d1d-4805-4ad6-9b4a-38b0e621affc/cni-plugins/0.log" Jul 15 21:49:58.135698 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.135680 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vchqc_ca0d0d1d-4805-4ad6-9b4a-38b0e621affc/bond-cni-plugin/0.log" Jul 15 21:49:58.155794 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.155773 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vchqc_ca0d0d1d-4805-4ad6-9b4a-38b0e621affc/routeoverride-cni/0.log" Jul 15 21:49:58.175807 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.175784 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vchqc_ca0d0d1d-4805-4ad6-9b4a-38b0e621affc/whereabouts-cni-bincopy/0.log" Jul 15 21:49:58.194200 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.194179 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vchqc_ca0d0d1d-4805-4ad6-9b4a-38b0e621affc/whereabouts-cni/0.log" Jul 15 21:49:58.280969 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.280946 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rr7c2_8dd406be-56be-4aa7-a84f-56fb3308fc02/kube-multus/0.log" Jul 15 21:49:58.419296 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.419220 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8cnsf_bb98c88c-8490-4747-ac99-546a6159a8d3/network-metrics-daemon/0.log" Jul 15 21:49:58.438208 ip-10-0-136-220 kubenswrapper[2553]: I0715 21:49:58.438185 2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8cnsf_bb98c88c-8490-4747-ac99-546a6159a8d3/kube-rbac-proxy/0.log"