Jul 11 23:31:02.871638 ip-10-0-137-251 systemd[1]: Starting Kubernetes Kubelet... Jul 11 23:31:04.587950 ip-10-0-137-251 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 11 23:31:04.587950 ip-10-0-137-251 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 11 23:31:04.587950 ip-10-0-137-251 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 11 23:31:04.587950 ip-10-0-137-251 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 11 23:31:04.587950 ip-10-0-137-251 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 11 23:31:04.589073 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.589007 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 11 23:31:04.593350 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593336 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 23:31:04.593350 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593350 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593354 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593357 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593361 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593364 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593368 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593371 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593374 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593377 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593380 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593382 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593385 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593387 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593390 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593393 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593396 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593398 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593400 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593403 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 23:31:04.593408 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593405 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593416 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593419 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593422 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593424 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593427 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593429 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593432 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593435 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593438 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593440 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593443 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593445 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593448 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593451 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593454 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593456 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593459 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593461 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 23:31:04.593865 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593464 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593466 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593469 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593472 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593474 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593477 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593479 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593482 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593484 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593487 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593489 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593494 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593509 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593513 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593516 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593520 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593524 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593528 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593531 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 23:31:04.594308 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593534 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593537 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593540 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593543 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593545 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593548 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593550 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593553 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593556 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593559 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593561 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593564 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593566 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593569 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593571 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593574 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593576 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593579 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593582 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593584 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 11 23:31:04.594788 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593587 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593590 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593592 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593595 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593597 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593600 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593602 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593605 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593608 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593611 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593614 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593616 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593619 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593621 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593623 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593626 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593628 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593631 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593633 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593636 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 23:31:04.595287 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.593638 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594336 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594343 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594347 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594351 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594354 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594357 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594359 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594362 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594365 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594367 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594371 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594375 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594377 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594380 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594383 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594385 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594387 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594390 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 23:31:04.595790 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594392 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594395 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594398 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594401 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594403 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594406 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594408 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594411 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594413 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594416 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594418 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594420 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594423 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594427 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594430 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594433 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594436 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594438 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594441 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 23:31:04.596253 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594443 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594446 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594448 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594452 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594454 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594457 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594459 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594462 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594464 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594468 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594471 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594473 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594476 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594479 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594482 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594485 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594488 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594491 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594494 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594497 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 23:31:04.596722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594514 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594518 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594521 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594523 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594526 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594529 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594532 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594535 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594537 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594540 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594542 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594545 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594547 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594550 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594552 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594555 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594557 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594562 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594564 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594567 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 23:31:04.597389 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594571 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594573 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594576 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594578 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594581 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594584 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594586 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594589 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594592 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594594 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594597 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594600 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594602 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594605 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594607 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594610 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594612 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594615 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594617 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594620 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 23:31:04.598037 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594622 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594624 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594992 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.594998 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595001 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595004 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595007 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595010 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595012 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595017 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595020 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595022 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595025 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595027 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595030 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595033 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595035 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595038 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 23:31:04.598571 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595041 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595044 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595046 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595049 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595052 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595055 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595058 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595060 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595063 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595066 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595068 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595071 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595073 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595076 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595078 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595081 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595083 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595086 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595088 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595090 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 23:31:04.598993 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595093 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595096 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595099 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595104 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595107 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595111 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595113 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595116 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595118 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595121 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595124 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595127 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595130 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595132 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595135 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595137 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595140 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595143 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595146 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595148 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 23:31:04.599468 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595150 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595153 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595156 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595158 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595161 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595163 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595166 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595168 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595170 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595173 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595175 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595178 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595180 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595183 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595185 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595189 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595191 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595195 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595197 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595200 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 23:31:04.599987 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595203 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595207 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595209 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595212 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595215 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595218 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595221 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595223 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595226 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595228 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595231 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595234 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595236 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595239 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595241 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595244 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595246 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595248 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595251 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595253 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 23:31:04.600518 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595256 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595259 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595263 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595321 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595327 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595333 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595336 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595342 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595345 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595349 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595353 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595356 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595359 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595362 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595365 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595369 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595372 2566 flags.go:64] FLAG: --cgroup-root="" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595374 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595377 2566 flags.go:64] FLAG: --client-ca-file="" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595380 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595383 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595387 2566 flags.go:64] FLAG: --cluster-domain="" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595390 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 11 23:31:04.600992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595393 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595396 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595400 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595403 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595406 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595409 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595412 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595415 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595418 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595421 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595424 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595427 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595430 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595433 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595436 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595439 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595443 2566 flags.go:64] FLAG: --enable-server="true" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595447 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595451 2566 flags.go:64] FLAG: --event-burst="100" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595453 2566 flags.go:64] FLAG: --event-qps="50" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595456 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595459 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595462 2566 flags.go:64] FLAG: --eviction-hard="" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595465 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 11 23:31:04.601603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595469 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595472 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595475 2566 flags.go:64] FLAG: --eviction-soft="" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595478 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595481 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595484 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595486 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595489 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595492 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595496 2566 flags.go:64] FLAG: --feature-gates="" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595513 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595516 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595519 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595523 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595526 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595529 2566 flags.go:64] FLAG: --help="false" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595532 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595536 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595539 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595542 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595545 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595548 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595551 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 11 23:31:04.602208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595554 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595559 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595561 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595564 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595568 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595571 2566 flags.go:64] FLAG: --kube-reserved="" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595574 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595576 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595579 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595584 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595587 2566 flags.go:64] FLAG: --lock-file="" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595590 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595593 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595596 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595601 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595604 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595607 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595610 2566 flags.go:64] FLAG: --logging-format="text" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595612 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595616 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595619 2566 flags.go:64] FLAG: --manifest-url="" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595622 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595626 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595629 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595636 2566 flags.go:64] FLAG: --max-pods="110" Jul 11 23:31:04.602834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595639 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595642 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595645 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595648 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595651 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595654 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595657 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595663 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595668 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595671 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595674 2566 flags.go:64] FLAG: --pod-cidr="" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595676 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595681 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595684 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595687 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595690 2566 flags.go:64] FLAG: --port="10250" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595693 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595699 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01b361d40d0659cf4" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595702 2566 flags.go:64] FLAG: --qos-reserved="" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595705 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595708 2566 flags.go:64] FLAG: --register-node="true" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595711 2566 flags.go:64] FLAG: --register-with-taints="" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595714 2566 flags.go:64] FLAG: --registry-burst="10" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595717 2566 flags.go:64] FLAG: --registry-qps="5" Jul 11 23:31:04.603435 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595720 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595723 2566 flags.go:64] FLAG: --reserved-memory="" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595726 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595730 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595733 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595736 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595739 2566 flags.go:64] FLAG: --runonce="false" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595742 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595745 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595748 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595750 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595753 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595756 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595759 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595762 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595765 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595769 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595772 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595775 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595778 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595781 2566 flags.go:64] FLAG: --system-cgroups="" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595784 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595789 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595792 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595795 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 11 23:31:04.604030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595800 2566 flags.go:64] FLAG: --tls-min-version="" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595803 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595806 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595809 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595812 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595815 2566 flags.go:64] FLAG: --v="2" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595819 2566 flags.go:64] FLAG: --version="false" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595823 2566 flags.go:64] FLAG: --vmodule="" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595826 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.595830 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595932 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595936 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595939 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595942 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595945 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595948 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595951 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595953 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595956 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595959 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595962 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595964 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 23:31:04.604645 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595967 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595970 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595973 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595975 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595977 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595980 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595982 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595985 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595988 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595990 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595995 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.595997 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596000 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596002 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596004 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596007 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596009 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596012 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596015 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 23:31:04.605188 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596018 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596020 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596023 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596026 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596030 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596034 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596037 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596039 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596043 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596047 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596050 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596053 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596055 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596059 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596062 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596065 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596067 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596070 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 23:31:04.605713 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596072 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596075 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596077 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596080 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596082 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596086 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596089 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596092 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596094 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596097 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596100 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596103 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596105 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596108 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596110 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596113 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596115 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596118 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596121 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596124 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 23:31:04.606171 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596127 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596129 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596132 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596134 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596137 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596139 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596142 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596144 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596147 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596149 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596151 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596154 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596157 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596159 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596161 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596164 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596166 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596170 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596173 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596176 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 23:31:04.606722 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596178 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596181 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596183 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596186 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596188 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596191 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596193 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596196 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596198 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.596201 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.596806 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.603404 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.603419 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.603437 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 11 23:31:04.607208 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.603444 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603515 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603520 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603524 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603527 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603530 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603533 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603536 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603539 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603542 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603545 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603547 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603550 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603555 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603558 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603562 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603565 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603568 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603571 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 23:31:04.607626 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603574 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603577 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603582 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603585 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603588 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603590 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603593 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603596 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603598 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603601 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603604 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603607 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603610 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603612 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603615 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603617 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603620 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603623 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603625 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603628 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 23:31:04.608079 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603630 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603633 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603636 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603638 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603641 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603644 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603646 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603649 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603652 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603655 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603657 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603660 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603663 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603665 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603668 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603670 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603673 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603675 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603678 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603680 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 23:31:04.608587 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603683 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603685 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603688 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603691 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603693 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603696 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603699 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603701 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603703 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603706 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603709 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603711 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603714 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603716 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603720 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603724 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603726 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603729 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603732 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 23:31:04.609103 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603734 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603737 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603743 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603746 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603748 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603751 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603754 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603756 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603759 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603762 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603764 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603766 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603769 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603773 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603777 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603780 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603783 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603786 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603788 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 23:31:04.609605 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603791 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603794 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603796 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.603801 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603929 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603934 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603937 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603940 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603943 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603946 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603948 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603951 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603954 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603956 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603959 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 11 23:31:04.610061 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603961 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603965 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603967 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603969 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603972 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603975 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603977 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603980 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603982 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603984 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603987 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603989 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603992 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603994 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.603997 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604000 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604002 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604005 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604007 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 11 23:31:04.610426 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604010 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604013 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604015 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604018 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604020 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604023 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604025 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604027 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604030 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604033 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604037 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604040 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604044 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604047 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604050 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604053 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604055 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604058 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604061 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 11 23:31:04.610902 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604063 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604066 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604068 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604070 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604073 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604075 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604078 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604080 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604083 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604085 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604088 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604090 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604092 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604095 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604097 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604100 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604104 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604107 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604109 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 11 23:31:04.611489 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604113 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604116 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604118 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604121 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604123 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604126 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604129 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604131 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604133 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604136 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604139 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604142 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604144 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604147 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604150 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604153 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604155 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604158 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604160 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604163 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 11 23:31:04.611973 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604165 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604168 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604170 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604173 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604175 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604177 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604180 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604182 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604185 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604187 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:04.604190 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.604195 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.604294 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.606123 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.607267 2566 server.go:1019] "Starting client certificate rotation" Jul 11 23:31:04.612461 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.607373 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 11 23:31:04.612895 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.607407 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 11 23:31:04.612895 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.609284 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 11 23:31:04.613374 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.613357 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 11 23:31:04.621374 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.621357 2566 log.go:25] "Validated CRI v1 runtime API" Jul 11 23:31:04.623717 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.623700 2566 log.go:25] "Validated CRI v1 image API" Jul 11 23:31:04.624609 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.624593 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 11 23:31:04.627352 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.627331 2566 fs.go:135] Filesystem UUIDs: map[1dcfc450-489f-4c7d-afa4-840fb8e9e685:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f844e076-08ee-46ae-bd8b-79129744c152:/dev/nvme0n1p3] Jul 11 23:31:04.627424 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.627350 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 11 23:31:04.633021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.632901 2566 manager.go:216] Machine: {Timestamp:2026-07-11 23:31:04.631735472 +0000 UTC m=+0.287358738 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100236 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20da86bdf4c98bfc64983e76eb2c00 SystemUUID:ec20da86-bdf4-c98b-fc64-983e76eb2c00 BootID:3994eaea-dbf9-4f37-8372-27698515542e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:83:21:48:14:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:83:21:48:14:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:ea:fb:ce:d1:cd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 11 23:31:04.633021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.633013 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 11 23:31:04.633174 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.633104 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 11 23:31:04.634697 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.634680 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 11 23:31:04.635576 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.635553 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 11 23:31:04.635708 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.635579 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 11 23:31:04.636199 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.636189 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 11 23:31:04.636199 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.636199 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 11 23:31:04.636273 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.636206 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 11 23:31:04.636899 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.636889 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 11 23:31:04.637562 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.637552 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 11 23:31:04.640908 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.640896 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 11 23:31:04.641026 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641017 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 11 23:31:04.641064 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641045 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 11 23:31:04.641064 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641055 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 11 23:31:04.641127 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641067 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 11 23:31:04.641127 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641076 2566 kubelet.go:398] "Adding apiserver pod source" Jul 11 23:31:04.641127 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641084 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 11 23:31:04.641370 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641358 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 11 23:31:04.641401 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.641381 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 11 23:31:04.643243 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.643220 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 11 23:31:04.644376 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.644361 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 11 23:31:04.644434 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.644391 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 11 23:31:04.645403 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645391 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645409 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645414 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645419 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645428 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645433 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645441 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645447 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645453 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645458 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 11 23:31:04.645464 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645467 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 11 23:31:04.645753 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.645473 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 11 23:31:04.646369 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.646359 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 11 23:31:04.646369 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.646369 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 11 23:31:04.649167 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.649152 2566 server.go:1268] "Started kubelet" Jul 11 23:31:04.649270 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.649240 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 11 23:31:04.649346 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.649252 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 11 23:31:04.649346 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.649330 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 11 23:31:04.649831 ip-10-0-137-251 systemd[1]: Started Kubernetes Kubelet. Jul 11 23:31:04.651766 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.651750 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 11 23:31:04.652752 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.652732 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 11 23:31:04.655272 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.655256 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 11 23:31:04.655772 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.655751 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 11 23:31:04.655772 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.655765 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 11 23:31:04.657251 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.656706 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 11 23:31:04.657251 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.656649 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 11 23:31:04.657251 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.656801 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 11 23:31:04.657251 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.656944 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 11 23:31:04.657251 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.656951 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 11 23:31:04.657251 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.656980 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:04.657996 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.657958 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 11 23:31:04.658116 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658012 2566 factory.go:153] Registering CRI-O factory Jul 11 23:31:04.658116 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658023 2566 factory.go:223] Registration of the crio container factory successfully Jul 11 23:31:04.658116 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658112 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 11 23:31:04.658274 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658130 2566 factory.go:55] Registering systemd factory Jul 11 23:31:04.658274 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658138 2566 factory.go:223] Registration of the systemd container factory successfully Jul 11 23:31:04.658274 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658158 2566 factory.go:103] Registering Raw factory Jul 11 23:31:04.658274 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658197 2566 manager.go:1196] Started watching for new ooms in manager Jul 11 23:31:04.658791 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.657844 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-251.ec2.internal.18c1602ade411202 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-251.ec2.internal,UID:ip-10-0-137-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-251.ec2.internal,},FirstTimestamp:2026-07-11 23:31:04.649126402 +0000 UTC m=+0.304749645,LastTimestamp:2026-07-11 23:31:04.649126402 +0000 UTC m=+0.304749645,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-251.ec2.internal,}" Jul 11 23:31:04.658791 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.658763 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 11 23:31:04.658941 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.658825 2566 manager.go:349] Starting recovery of all containers Jul 11 23:31:04.658941 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.658900 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 11 23:31:04.660540 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.660320 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 11 23:31:04.660540 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.660441 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 11 23:31:04.671117 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.671013 2566 manager.go:354] Recovery completed Jul 11 23:31:04.675482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.675469 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 23:31:04.677345 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.677314 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientMemory" Jul 11 23:31:04.677406 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.677359 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 23:31:04.677406 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.677369 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientPID" Jul 11 23:31:04.677867 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.677853 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 23:31:04.677971 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.677852 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 11 23:31:04.677971 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.677925 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 11 23:31:04.677971 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.677944 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 11 23:31:04.678516 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.678490 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientMemory" Jul 11 23:31:04.678560 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.678530 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 23:31:04.678560 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.678540 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientPID" Jul 11 23:31:04.679004 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.678927 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-251.ec2.internal.18c1602adfefa944 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-251.ec2.internal,UID:ip-10-0-137-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-251.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-251.ec2.internal,},FirstTimestamp:2026-07-11 23:31:04.677345604 +0000 UTC m=+0.332968848,LastTimestamp:2026-07-11 23:31:04.677345604 +0000 UTC m=+0.332968848,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-251.ec2.internal,}" Jul 11 23:31:04.680096 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.680085 2566 policy_none.go:49] "None policy: Start" Jul 11 23:31:04.680153 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.680099 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 11 23:31:04.680153 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.680110 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 11 23:31:04.680830 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.680811 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9mt8d" Jul 11 23:31:04.682019 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.682008 2566 policy_none.go:47] "Start" Jul 11 23:31:04.687867 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.687850 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9mt8d" Jul 11 23:31:04.688819 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.688759 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-251.ec2.internal.18c1602adfeff079 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-251.ec2.internal,UID:ip-10-0-137-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-251.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-251.ec2.internal,},FirstTimestamp:2026-07-11 23:31:04.677363833 +0000 UTC m=+0.332987078,LastTimestamp:2026-07-11 23:31:04.677363833 +0000 UTC m=+0.332987078,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-251.ec2.internal,}" Jul 11 23:31:04.730035 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.730023 2566 manager.go:337] "Starting Device Plugin manager" Jul 11 23:31:04.730103 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.730055 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 11 23:31:04.730103 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.730068 2566 server.go:87] "Starting device plugin registration server" Jul 11 23:31:04.730320 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.730308 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 11 23:31:04.730359 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.730342 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 11 23:31:04.730487 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.730421 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 11 23:31:04.730487 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.730517 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 11 23:31:04.730628 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.730527 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 11 23:31:04.730885 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.730861 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 11 23:31:04.730961 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.730918 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:04.799736 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.799711 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 11 23:31:04.801604 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.800824 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 11 23:31:04.801604 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.800856 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 11 23:31:04.801604 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.800873 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 11 23:31:04.801604 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.800903 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 11 23:31:04.802774 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.802754 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 23:31:04.831334 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.831310 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 23:31:04.832710 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.832695 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientMemory" Jul 11 23:31:04.832778 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.832725 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 23:31:04.832778 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.832734 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientPID" Jul 11 23:31:04.832778 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.832757 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.840927 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.840912 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.840977 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.840931 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-251.ec2.internal\": node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:04.876013 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.875994 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:04.901156 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.901138 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal"] Jul 11 23:31:04.901212 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.901189 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 23:31:04.902859 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.902845 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientMemory" Jul 11 23:31:04.902921 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.902871 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 23:31:04.902921 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.902880 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientPID" Jul 11 23:31:04.904902 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.904890 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 23:31:04.905050 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905035 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.905098 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905068 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 23:31:04.905622 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905607 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientMemory" Jul 11 23:31:04.905709 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905631 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 23:31:04.905709 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905641 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientPID" Jul 11 23:31:04.905709 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905603 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientMemory" Jul 11 23:31:04.905709 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905677 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 23:31:04.905709 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.905696 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientPID" Jul 11 23:31:04.908594 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.908577 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.908667 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.908606 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 11 23:31:04.909233 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.909218 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientMemory" Jul 11 23:31:04.909324 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.909246 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 11 23:31:04.909324 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.909259 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeHasSufficientPID" Jul 11 23:31:04.926092 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.926070 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-251.ec2.internal\" not found" node="ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.930584 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.930566 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-251.ec2.internal\" not found" node="ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.959094 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.959078 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/554e1624a4d584e0be3ea6067bec89c5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-251.ec2.internal\" (UID: \"554e1624a4d584e0be3ea6067bec89c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.959168 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.959100 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b8a28ec8a6e788a6d3d86a8cc3699c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal\" (UID: \"6b8a28ec8a6e788a6d3d86a8cc3699c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.959168 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:04.959118 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b8a28ec8a6e788a6d3d86a8cc3699c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal\" (UID: \"6b8a28ec8a6e788a6d3d86a8cc3699c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:04.976714 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:04.976677 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:05.059665 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.059643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/554e1624a4d584e0be3ea6067bec89c5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-251.ec2.internal\" (UID: \"554e1624a4d584e0be3ea6067bec89c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.059737 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.059682 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b8a28ec8a6e788a6d3d86a8cc3699c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal\" (UID: \"6b8a28ec8a6e788a6d3d86a8cc3699c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.059737 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.059698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b8a28ec8a6e788a6d3d86a8cc3699c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal\" (UID: \"6b8a28ec8a6e788a6d3d86a8cc3699c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.059799 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.059746 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b8a28ec8a6e788a6d3d86a8cc3699c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal\" (UID: \"6b8a28ec8a6e788a6d3d86a8cc3699c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.059799 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.059752 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/554e1624a4d584e0be3ea6067bec89c5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-251.ec2.internal\" (UID: \"554e1624a4d584e0be3ea6067bec89c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.059799 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.059787 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b8a28ec8a6e788a6d3d86a8cc3699c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal\" (UID: \"6b8a28ec8a6e788a6d3d86a8cc3699c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.077764 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:05.077747 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:05.178552 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:05.178528 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:05.228724 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.228684 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.233127 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.232967 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.278620 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:05.278598 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:05.379092 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:05.379067 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:05.479649 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:05.479602 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:05.580096 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:05.580058 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-251.ec2.internal\" not found" Jul 11 23:31:05.607239 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.607216 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 11 23:31:05.607705 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.607345 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 11 23:31:05.641646 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.641626 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 23:31:05.655861 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.655841 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 11 23:31:05.656007 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.655990 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.666826 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.666807 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 11 23:31:05.669666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.669638 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 11 23:31:05.671144 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.671129 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" Jul 11 23:31:05.679409 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.679393 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 11 23:31:05.685361 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.685344 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gw7x4" Jul 11 23:31:05.692459 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.692435 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-10 23:26:04 +0000 UTC" deadline="2028-02-06 09:14:57.322785644 +0000 UTC" Jul 11 23:31:05.692459 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.692458 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13785h43m51.630330664s" Jul 11 23:31:05.693195 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.693180 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gw7x4" Jul 11 23:31:05.752811 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:05.752786 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8a28ec8a6e788a6d3d86a8cc3699c9.slice/crio-3e4017b3498cc383d6fc6b5bdb93530a0ed52259f77d80813a4ae6ce928e0001 WatchSource:0}: Error finding container 3e4017b3498cc383d6fc6b5bdb93530a0ed52259f77d80813a4ae6ce928e0001: Status 404 returned error can't find the container with id 3e4017b3498cc383d6fc6b5bdb93530a0ed52259f77d80813a4ae6ce928e0001 Jul 11 23:31:05.753375 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:05.753355 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554e1624a4d584e0be3ea6067bec89c5.slice/crio-88addc635662fb1a7686ba84ffee45807dbf839805118e5fe509621c197be985 WatchSource:0}: Error finding container 88addc635662fb1a7686ba84ffee45807dbf839805118e5fe509621c197be985: Status 404 returned error can't find the container with id 88addc635662fb1a7686ba84ffee45807dbf839805118e5fe509621c197be985 Jul 11 23:31:05.758818 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.758802 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 11 23:31:05.760623 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.760607 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 23:31:05.803443 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.803404 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" event={"ID":"554e1624a4d584e0be3ea6067bec89c5","Type":"ContainerStarted","Data":"88addc635662fb1a7686ba84ffee45807dbf839805118e5fe509621c197be985"} Jul 11 23:31:05.804430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.804411 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" event={"ID":"6b8a28ec8a6e788a6d3d86a8cc3699c9","Type":"ContainerStarted","Data":"3e4017b3498cc383d6fc6b5bdb93530a0ed52259f77d80813a4ae6ce928e0001"} Jul 11 23:31:05.961194 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:05.961173 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 23:31:06.642221 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.642193 2566 apiserver.go:52] "Watching apiserver" Jul 11 23:31:06.652983 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.652962 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 11 23:31:06.653357 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.653336 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-q5tgg","kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal","openshift-multus/multus-additional-cni-plugins-gzbk4","openshift-network-diagnostics/network-check-target-66kvw","openshift-network-operator/iptables-alerter-2bwwc","openshift-ovn-kubernetes/ovnkube-node-tvxms","kube-system/konnectivity-agent-qbjd9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs","openshift-cluster-node-tuning-operator/tuned-k4qzh","openshift-dns/node-resolver-f58r4","openshift-image-registry/node-ca-lwcsq","openshift-multus/multus-phdpk","openshift-multus/network-metrics-daemon-2jq6z"] Jul 11 23:31:06.655828 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.655803 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.658043 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.658027 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:06.658127 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.658086 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:06.658377 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.658355 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ddvdx\"" Jul 11 23:31:06.658474 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.658418 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 11 23:31:06.658474 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.658434 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:31:06.658474 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.658360 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 11 23:31:06.660597 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.660427 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.663410 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.663221 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 11 23:31:06.663410 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.663265 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-w2rkq\"" Jul 11 23:31:06.663410 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.663338 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 11 23:31:06.663614 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.663440 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 11 23:31:06.663614 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.663578 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 11 23:31:06.663835 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.663820 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.666063 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.666043 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.666180 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.666163 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 11 23:31:06.666667 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.666648 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zs5np\"" Jul 11 23:31:06.666860 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.666844 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 11 23:31:06.666860 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.666852 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 11 23:31:06.668175 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.668154 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc515d92-277f-408f-af90-7c8561df5b17-iptables-alerter-script\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.668261 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.668181 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc515d92-277f-408f-af90-7c8561df5b17-host-slash\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.668261 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.668198 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsv9z\" (UniqueName: \"kubernetes.io/projected/dc515d92-277f-408f-af90-7c8561df5b17-kube-api-access-qsv9z\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.668261 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.668213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:06.668261 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.668231 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88nhd\" (UniqueName: \"kubernetes.io/projected/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-kube-api-access-88nhd\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:06.668424 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.668408 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.668849 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.668830 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 11 23:31:06.670190 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670169 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 11 23:31:06.670297 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670279 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 11 23:31:06.670354 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670172 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 11 23:31:06.670587 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670523 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 11 23:31:06.670587 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670527 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6gg67\"" Jul 11 23:31:06.670587 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670569 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 11 23:31:06.670766 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670709 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:06.670817 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.670767 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:06.670952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670882 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 11 23:31:06.670952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670882 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 11 23:31:06.671095 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.670991 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gt2f5\"" Jul 11 23:31:06.672939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.672919 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:06.675175 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.675159 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 11 23:31:06.675439 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.675421 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vhqzf\"" Jul 11 23:31:06.675539 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.675429 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 11 23:31:06.675539 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.675490 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.677998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.677979 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.678087 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.678062 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 11 23:31:06.678194 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.678174 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 11 23:31:06.678267 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.678250 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 11 23:31:06.678467 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.678451 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tc9wg\"" Jul 11 23:31:06.680413 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.680313 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.680936 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.680916 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:31:06.681034 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.681015 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wxdp6\"" Jul 11 23:31:06.681085 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.681015 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 11 23:31:06.682686 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.682657 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.682777 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.682738 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:06.682862 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.682841 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 11 23:31:06.682918 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.682890 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 11 23:31:06.683038 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.682990 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5nn7w\"" Jul 11 23:31:06.696290 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.696265 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-10 23:26:05 +0000 UTC" deadline="2028-01-18 04:05:52.734242114 +0000 UTC" Jul 11 23:31:06.696381 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.696291 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13324h34m46.037954813s" Jul 11 23:31:06.757815 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.757787 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 11 23:31:06.769296 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769272 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.769404 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc515d92-277f-408f-af90-7c8561df5b17-iptables-alerter-script\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.769404 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-os-release\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.769404 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-run\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.769404 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a9629d-d41e-49b5-b3d1-51140debf751-tmp\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-kubelet\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769449 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769476 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-system-cni-dir\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769513 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-registration-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769537 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-device-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769560 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67a00afc-3605-4f51-9303-12299c30fd84-hosts-file\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769585 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.769593 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:06.769632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-cni-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.769694 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs podName:67e8fbc2-1ffc-42d1-8ea6-307da940eda6 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:07.269647116 +0000 UTC m=+2.925270368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs") pod "network-metrics-daemon-2jq6z" (UID: "67e8fbc2-1ffc-42d1-8ea6-307da940eda6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769727 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-socket-dir-parent\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769757 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-netns\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769783 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovnkube-config\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769815 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cnibin\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769839 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95d3c2ae-d983-46de-a4fd-a417d0059a37-serviceca\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769890 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2d5c\" (UniqueName: \"kubernetes.io/projected/a8a9629d-d41e-49b5-b3d1-51140debf751-kube-api-access-b2d5c\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80391286-a001-4714-95f7-47d673ca1a6b-kubelet-config\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.769988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-system-cni-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770000 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dc515d92-277f-408f-af90-7c8561df5b17-iptables-alerter-script\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.770036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770024 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6d75\" (UniqueName: \"kubernetes.io/projected/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-kube-api-access-l6d75\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770054 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-log-socket\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysconfig\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770091 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-lib-modules\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-host\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770123 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80391286-a001-4714-95f7-47d673ca1a6b-dbus\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-cnibin\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770149 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-cni-binary-copy\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770162 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-etc-kubernetes\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770189 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d87e378d-6b17-43a7-8667-e095db0b08b0-agent-certs\") pod \"konnectivity-agent-qbjd9\" (UID: \"d87e378d-6b17-43a7-8667-e095db0b08b0\") " pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770208 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-k8s-cni-cncf-io\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770243 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-systemd-units\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770274 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-ovn\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770298 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovn-node-metrics-cert\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770323 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770352 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-cni-multus\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770375 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-kubelet\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.770540 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-hostroot\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770418 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-multus-certs\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-node-log\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ghls\" (UniqueName: \"kubernetes.io/projected/95d3c2ae-d983-46de-a4fd-a417d0059a37-kube-api-access-7ghls\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770515 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-var-lib-kubelet\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-cni-netd\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770621 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a00afc-3605-4f51-9303-12299c30fd84-tmp-dir\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysctl-conf\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770664 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-run-netns\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770691 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770718 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770754 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc515d92-277f-408f-af90-7c8561df5b17-host-slash\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsv9z\" (UniqueName: \"kubernetes.io/projected/dc515d92-277f-408f-af90-7c8561df5b17-kube-api-access-qsv9z\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88nhd\" (UniqueName: \"kubernetes.io/projected/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-kube-api-access-88nhd\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-sys-fs\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.771293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.770958 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dc515d92-277f-408f-af90-7c8561df5b17-host-slash\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771007 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysctl-d\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-systemd\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-env-overrides\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771128 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-kubernetes\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-tuned\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-cni-bin\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771207 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-var-lib-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771238 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-run-ovn-kubernetes\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771274 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovnkube-script-lib\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z78xl\" (UniqueName: \"kubernetes.io/projected/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-kube-api-access-z78xl\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771319 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-socket-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771344 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knbv9\" (UniqueName: \"kubernetes.io/projected/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-kube-api-access-knbv9\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-systemd\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771392 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-daemon-config\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771416 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.771984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771445 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-cni-bin\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771467 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-modprobe-d\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-conf-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771548 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-os-release\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771584 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-slash\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771621 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-etc-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771649 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c9dm\" (UniqueName: \"kubernetes.io/projected/67a00afc-3605-4f51-9303-12299c30fd84-kube-api-access-7c9dm\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771664 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95d3c2ae-d983-46de-a4fd-a417d0059a37-host\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771684 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4m99\" (UniqueName: \"kubernetes.io/projected/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-kube-api-access-z4m99\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771709 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d87e378d-6b17-43a7-8667-e095db0b08b0-konnectivity-ca\") pod \"konnectivity-agent-qbjd9\" (UID: \"d87e378d-6b17-43a7-8667-e095db0b08b0\") " pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.772468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.771737 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-sys\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.777207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.777186 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 11 23:31:06.780529 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.780489 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88nhd\" (UniqueName: \"kubernetes.io/projected/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-kube-api-access-88nhd\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:06.780529 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.780514 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsv9z\" (UniqueName: \"kubernetes.io/projected/dc515d92-277f-408f-af90-7c8561df5b17-kube-api-access-qsv9z\") pod \"iptables-alerter-2bwwc\" (UID: \"dc515d92-277f-408f-af90-7c8561df5b17\") " pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.871956 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.871927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2d5c\" (UniqueName: \"kubernetes.io/projected/a8a9629d-d41e-49b5-b3d1-51140debf751-kube-api-access-b2d5c\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.871956 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.871955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80391286-a001-4714-95f7-47d673ca1a6b-kubelet-config\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.871971 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.871991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-system-cni-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872014 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6d75\" (UniqueName: \"kubernetes.io/projected/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-kube-api-access-l6d75\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872035 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-log-socket\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysconfig\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-lib-modules\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872093 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-log-socket\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.872140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-host\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872146 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysconfig\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-host\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.872117 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872057 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80391286-a001-4714-95f7-47d673ca1a6b-kubelet-config\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80391286-a001-4714-95f7-47d673ca1a6b-dbus\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.872240 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret podName:80391286-a001-4714-95f7-47d673ca1a6b nodeName:}" failed. No retries permitted until 2026-07-11 23:31:07.372220964 +0000 UTC m=+3.027844213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret") pod "global-pull-secret-syncer-q5tgg" (UID: "80391286-a001-4714-95f7-47d673ca1a6b") : object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872241 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-lib-modules\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872277 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80391286-a001-4714-95f7-47d673ca1a6b-dbus\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872281 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-system-cni-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.872490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-cnibin\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.872956 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-cni-binary-copy\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.872956 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.872819 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-cnibin\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873024 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-etc-kubernetes\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873170 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-etc-kubernetes\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d87e378d-6b17-43a7-8667-e095db0b08b0-agent-certs\") pod \"konnectivity-agent-qbjd9\" (UID: \"d87e378d-6b17-43a7-8667-e095db0b08b0\") " pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-k8s-cni-cncf-io\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873334 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-systemd-units\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-ovn\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873378 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovn-node-metrics-cert\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-k8s-cni-cncf-io\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873539 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-cni-multus\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873583 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-kubelet\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.873605 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873612 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-hostroot\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-multus-certs\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-node-log\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873701 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ghls\" (UniqueName: \"kubernetes.io/projected/95d3c2ae-d983-46de-a4fd-a417d0059a37-kube-api-access-7ghls\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873768 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-var-lib-kubelet\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873802 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-cni-netd\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873834 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a00afc-3605-4f51-9303-12299c30fd84-tmp-dir\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysctl-conf\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-run-netns\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.873993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-sys-fs\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874025 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysctl-d\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874056 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-systemd\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874090 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-env-overrides\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.874281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874121 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-kubernetes\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874145 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-tuned\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874661 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-cni-bin\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874697 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-var-lib-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874730 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-run-ovn-kubernetes\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovnkube-script-lib\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874798 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z78xl\" (UniqueName: \"kubernetes.io/projected/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-kube-api-access-z78xl\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-socket-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874860 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knbv9\" (UniqueName: \"kubernetes.io/projected/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-kube-api-access-knbv9\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874893 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-systemd\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-daemon-config\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874959 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.874987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-cni-bin\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875091 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-run-netns\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875100 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-node-log\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875172 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875283 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysctl-d\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.875662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-run-ovn-kubernetes\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875525 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-cni-bin\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875920 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875824 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-env-overrides\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.875920 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.875898 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-systemd-units\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876016 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-socket-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876242 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovnkube-script-lib\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-systemd\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876419 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cni-binary-copy\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876459 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876474 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876563 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-systemd\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876568 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-sys-fs\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876613 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-cni-multus\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876650 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-cni-binary-copy\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876665 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-cni-netd\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876681 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-cni-bin\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876725 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876751 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-hostroot\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876774 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-var-lib-kubelet\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876783 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a00afc-3605-4f51-9303-12299c30fd84-tmp-dir\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876781 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-multus-certs\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.877097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876799 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-var-lib-kubelet\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-modprobe-d\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876898 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-kubernetes\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876907 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-sysctl-conf\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876927 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-run-ovn\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876957 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-var-lib-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.876973 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-daemon-config\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-modprobe-d\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877109 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-conf-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877159 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-os-release\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-slash\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877240 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-conf-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877237 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-slash\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-etc-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877319 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c9dm\" (UniqueName: \"kubernetes.io/projected/67a00afc-3605-4f51-9303-12299c30fd84-kube-api-access-7c9dm\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877336 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-os-release\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95d3c2ae-d983-46de-a4fd-a417d0059a37-host\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-etc-openvswitch\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.878405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877416 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/95d3c2ae-d983-46de-a4fd-a417d0059a37-host\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4m99\" (UniqueName: \"kubernetes.io/projected/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-kube-api-access-z4m99\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877479 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d87e378d-6b17-43a7-8667-e095db0b08b0-konnectivity-ca\") pod \"konnectivity-agent-qbjd9\" (UID: \"d87e378d-6b17-43a7-8667-e095db0b08b0\") " pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877621 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-sys\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877731 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-sys\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877756 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a8a9629d-d41e-49b5-b3d1-51140debf751-etc-tuned\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877884 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877987 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d87e378d-6b17-43a7-8667-e095db0b08b0-agent-certs\") pod \"konnectivity-agent-qbjd9\" (UID: \"d87e378d-6b17-43a7-8667-e095db0b08b0\") " pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.877993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-os-release\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-run\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878089 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a9629d-d41e-49b5-b3d1-51140debf751-tmp\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878127 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-kubelet\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878093 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a8a9629d-d41e-49b5-b3d1-51140debf751-run\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878441 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-system-cni-dir\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.880282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878482 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-registration-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-device-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878540 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-system-cni-dir\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67a00afc-3605-4f51-9303-12299c30fd84-hosts-file\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878573 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-os-release\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878600 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-registration-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-device-dir\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878676 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-host-kubelet\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878680 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-cni-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878709 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67a00afc-3605-4f51-9303-12299c30fd84-hosts-file\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878719 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-socket-dir-parent\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878743 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-cni-dir\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d87e378d-6b17-43a7-8667-e095db0b08b0-konnectivity-ca\") pod \"konnectivity-agent-qbjd9\" (UID: \"d87e378d-6b17-43a7-8667-e095db0b08b0\") " pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878843 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-netns\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878853 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-multus-socket-dir-parent\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovnkube-config\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.881109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878952 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cnibin\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.878995 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.879038 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-host-run-netns\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.879165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95d3c2ae-d983-46de-a4fd-a417d0059a37-serviceca\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.879209 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-cnibin\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.879295 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.879467 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovnkube-config\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.880815 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/95d3c2ae-d983-46de-a4fd-a417d0059a37-serviceca\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.880974 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-ovn-node-metrics-cert\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.881189 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2d5c\" (UniqueName: \"kubernetes.io/projected/a8a9629d-d41e-49b5-b3d1-51140debf751-kube-api-access-b2d5c\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.881952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.881431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6d75\" (UniqueName: \"kubernetes.io/projected/941ffd9d-f5b2-4c5c-837b-e8bf8e26481b-kube-api-access-l6d75\") pod \"multus-phdpk\" (UID: \"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b\") " pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.882759 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.882695 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8a9629d-d41e-49b5-b3d1-51140debf751-tmp\") pod \"tuned-k4qzh\" (UID: \"a8a9629d-d41e-49b5-b3d1-51140debf751\") " pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:06.883590 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.883561 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knbv9\" (UniqueName: \"kubernetes.io/projected/957ca13a-7a50-4c5c-9f09-d4bb32adb8cd-kube-api-access-knbv9\") pod \"aws-ebs-csi-driver-node-m2qzs\" (UID: \"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:06.884211 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.884189 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z78xl\" (UniqueName: \"kubernetes.io/projected/d77fef30-62dd-4edf-b2c9-3d14a4a1a614-kube-api-access-z78xl\") pod \"ovnkube-node-tvxms\" (UID: \"d77fef30-62dd-4edf-b2c9-3d14a4a1a614\") " pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.884211 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.884202 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 23:31:06.884494 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.884221 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 23:31:06.884494 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.884233 2566 projected.go:196] Error preparing data for projected volume kube-api-access-nhm4x for pod openshift-network-diagnostics/network-check-target-66kvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:06.884494 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:06.884302 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x podName:2162ffaf-d5ff-4e93-bc07-02a8db8d7caf nodeName:}" failed. No retries permitted until 2026-07-11 23:31:07.384284259 +0000 UTC m=+3.039907506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nhm4x" (UniqueName: "kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x") pod "network-check-target-66kvw" (UID: "2162ffaf-d5ff-4e93-bc07-02a8db8d7caf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:06.884494 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.884444 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ghls\" (UniqueName: \"kubernetes.io/projected/95d3c2ae-d983-46de-a4fd-a417d0059a37-kube-api-access-7ghls\") pod \"node-ca-lwcsq\" (UID: \"95d3c2ae-d983-46de-a4fd-a417d0059a37\") " pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.885170 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.885153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c9dm\" (UniqueName: \"kubernetes.io/projected/67a00afc-3605-4f51-9303-12299c30fd84-kube-api-access-7c9dm\") pod \"node-resolver-f58r4\" (UID: \"67a00afc-3605-4f51-9303-12299c30fd84\") " pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:06.887351 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.887325 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4m99\" (UniqueName: \"kubernetes.io/projected/d08914c3-1c8f-4788-9b12-3cb52cb04ba9-kube-api-access-z4m99\") pod \"multus-additional-cni-plugins-gzbk4\" (UID: \"d08914c3-1c8f-4788-9b12-3cb52cb04ba9\") " pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:06.970128 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.970070 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2bwwc" Jul 11 23:31:06.976949 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.976928 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-phdpk" Jul 11 23:31:06.984910 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.984889 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lwcsq" Jul 11 23:31:06.989631 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.989614 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:06.996168 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:06.996151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" Jul 11 23:31:07.003155 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.003134 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:07.009635 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.009619 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" Jul 11 23:31:07.015165 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.015149 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" Jul 11 23:31:07.020686 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.020661 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-f58r4" Jul 11 23:31:07.119992 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.119971 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 11 23:31:07.282939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.282867 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:07.283062 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.283007 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:07.283107 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.283069 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs podName:67e8fbc2-1ffc-42d1-8ea6-307da940eda6 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:08.283053873 +0000 UTC m=+3.938677120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs") pod "network-metrics-daemon-2jq6z" (UID: "67e8fbc2-1ffc-42d1-8ea6-307da940eda6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:07.383363 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.383329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:07.383528 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.383486 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:07.383578 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.383562 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret podName:80391286-a001-4714-95f7-47d673ca1a6b nodeName:}" failed. No retries permitted until 2026-07-11 23:31:08.383547408 +0000 UTC m=+4.039170652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret") pod "global-pull-secret-syncer-q5tgg" (UID: "80391286-a001-4714-95f7-47d673ca1a6b") : object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:07.484070 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.484041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:07.484238 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.484200 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 23:31:07.484238 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.484218 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 23:31:07.484238 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.484233 2566 projected.go:196] Error preparing data for projected volume kube-api-access-nhm4x for pod openshift-network-diagnostics/network-check-target-66kvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:07.484403 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:07.484285 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x podName:2162ffaf-d5ff-4e93-bc07-02a8db8d7caf nodeName:}" failed. No retries permitted until 2026-07-11 23:31:08.484268781 +0000 UTC m=+4.139892030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhm4x" (UniqueName: "kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x") pod "network-check-target-66kvw" (UID: "2162ffaf-d5ff-4e93-bc07-02a8db8d7caf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:07.631224 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.631194 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957ca13a_7a50_4c5c_9f09_d4bb32adb8cd.slice/crio-09eba8722452cd2e5548bf4c320662a404271aa82e8c5f099520892c70993a20 WatchSource:0}: Error finding container 09eba8722452cd2e5548bf4c320662a404271aa82e8c5f099520892c70993a20: Status 404 returned error can't find the container with id 09eba8722452cd2e5548bf4c320662a404271aa82e8c5f099520892c70993a20 Jul 11 23:31:07.632209 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.632186 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a00afc_3605_4f51_9303_12299c30fd84.slice/crio-5b07876239e4fb2d896234d0b3f49d826d8e47f5505e0c537fc2719e51b351ba WatchSource:0}: Error finding container 5b07876239e4fb2d896234d0b3f49d826d8e47f5505e0c537fc2719e51b351ba: Status 404 returned error can't find the container with id 5b07876239e4fb2d896234d0b3f49d826d8e47f5505e0c537fc2719e51b351ba Jul 11 23:31:07.633225 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.633174 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc515d92_277f_408f_af90_7c8561df5b17.slice/crio-baf8363af5f405f1ad0dafb029b705f552059ece57705edd3723890633c2927e WatchSource:0}: Error finding container baf8363af5f405f1ad0dafb029b705f552059ece57705edd3723890633c2927e: Status 404 returned error can't find the container with id baf8363af5f405f1ad0dafb029b705f552059ece57705edd3723890633c2927e Jul 11 23:31:07.637030 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.636611 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87e378d_6b17_43a7_8667_e095db0b08b0.slice/crio-2f8c4e6d5e9c03bc39b414c2f185f649cb7e7fd116dfa3ae731cf03b10ae8b6b WatchSource:0}: Error finding container 2f8c4e6d5e9c03bc39b414c2f185f649cb7e7fd116dfa3ae731cf03b10ae8b6b: Status 404 returned error can't find the container with id 2f8c4e6d5e9c03bc39b414c2f185f649cb7e7fd116dfa3ae731cf03b10ae8b6b Jul 11 23:31:07.637923 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.637902 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941ffd9d_f5b2_4c5c_837b_e8bf8e26481b.slice/crio-67583ef669d40699f563b2af54b5b90350faf8c004832466afc367dfc21dd2c7 WatchSource:0}: Error finding container 67583ef669d40699f563b2af54b5b90350faf8c004832466afc367dfc21dd2c7: Status 404 returned error can't find the container with id 67583ef669d40699f563b2af54b5b90350faf8c004832466afc367dfc21dd2c7 Jul 11 23:31:07.638402 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.638386 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08914c3_1c8f_4788_9b12_3cb52cb04ba9.slice/crio-4bbd656b88b69ad1f1e177d98e10ae8760cb50e707b3858eaae777bec268c230 WatchSource:0}: Error finding container 4bbd656b88b69ad1f1e177d98e10ae8760cb50e707b3858eaae777bec268c230: Status 404 returned error can't find the container with id 4bbd656b88b69ad1f1e177d98e10ae8760cb50e707b3858eaae777bec268c230 Jul 11 23:31:07.660262 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.660239 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77fef30_62dd_4edf_b2c9_3d14a4a1a614.slice/crio-76cc3f4af7c02749935ddb00edb1596602e8f6da039fa1da7458016d85a9ae99 WatchSource:0}: Error finding container 76cc3f4af7c02749935ddb00edb1596602e8f6da039fa1da7458016d85a9ae99: Status 404 returned error can't find the container with id 76cc3f4af7c02749935ddb00edb1596602e8f6da039fa1da7458016d85a9ae99 Jul 11 23:31:07.660891 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.660858 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d3c2ae_d983_46de_a4fd_a417d0059a37.slice/crio-ce7000ce6a8d1a7efed9ef7324c48c60f7125a1bd81a39f3b5f06075408a566a WatchSource:0}: Error finding container ce7000ce6a8d1a7efed9ef7324c48c60f7125a1bd81a39f3b5f06075408a566a: Status 404 returned error can't find the container with id ce7000ce6a8d1a7efed9ef7324c48c60f7125a1bd81a39f3b5f06075408a566a Jul 11 23:31:07.661714 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:07.661667 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a9629d_d41e_49b5_b3d1_51140debf751.slice/crio-b9232443f5d1fcbb3b94bff0db7edcb6d881035854782f1814014bd9d46a7f77 WatchSource:0}: Error finding container b9232443f5d1fcbb3b94bff0db7edcb6d881035854782f1814014bd9d46a7f77: Status 404 returned error can't find the container with id b9232443f5d1fcbb3b94bff0db7edcb6d881035854782f1814014bd9d46a7f77 Jul 11 23:31:07.697021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.696864 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-10 23:26:05 +0000 UTC" deadline="2028-04-13 18:08:44.933262342 +0000 UTC" Jul 11 23:31:07.697093 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.697024 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15402h37m37.236244241s" Jul 11 23:31:07.808055 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.808026 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" event={"ID":"a8a9629d-d41e-49b5-b3d1-51140debf751","Type":"ContainerStarted","Data":"b9232443f5d1fcbb3b94bff0db7edcb6d881035854782f1814014bd9d46a7f77"} Jul 11 23:31:07.809129 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.809100 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"76cc3f4af7c02749935ddb00edb1596602e8f6da039fa1da7458016d85a9ae99"} Jul 11 23:31:07.810188 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.810169 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerStarted","Data":"4bbd656b88b69ad1f1e177d98e10ae8760cb50e707b3858eaae777bec268c230"} Jul 11 23:31:07.811177 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.811152 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qbjd9" event={"ID":"d87e378d-6b17-43a7-8667-e095db0b08b0","Type":"ContainerStarted","Data":"2f8c4e6d5e9c03bc39b414c2f185f649cb7e7fd116dfa3ae731cf03b10ae8b6b"} Jul 11 23:31:07.812202 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.812183 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f58r4" event={"ID":"67a00afc-3605-4f51-9303-12299c30fd84","Type":"ContainerStarted","Data":"5b07876239e4fb2d896234d0b3f49d826d8e47f5505e0c537fc2719e51b351ba"} Jul 11 23:31:07.813337 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.813319 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" event={"ID":"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd","Type":"ContainerStarted","Data":"09eba8722452cd2e5548bf4c320662a404271aa82e8c5f099520892c70993a20"} Jul 11 23:31:07.814472 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.814450 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lwcsq" event={"ID":"95d3c2ae-d983-46de-a4fd-a417d0059a37","Type":"ContainerStarted","Data":"ce7000ce6a8d1a7efed9ef7324c48c60f7125a1bd81a39f3b5f06075408a566a"} Jul 11 23:31:07.815650 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.815633 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phdpk" event={"ID":"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b","Type":"ContainerStarted","Data":"67583ef669d40699f563b2af54b5b90350faf8c004832466afc367dfc21dd2c7"} Jul 11 23:31:07.816624 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.816602 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bwwc" event={"ID":"dc515d92-277f-408f-af90-7c8561df5b17","Type":"ContainerStarted","Data":"baf8363af5f405f1ad0dafb029b705f552059ece57705edd3723890633c2927e"} Jul 11 23:31:07.818210 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:07.818192 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" event={"ID":"554e1624a4d584e0be3ea6067bec89c5","Type":"ContainerStarted","Data":"5c72d109aed5ca5ebc922e7153d8ecb91320a6b36c443b0af1307b104361295d"} Jul 11 23:31:08.290802 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.290767 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:08.290980 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.290920 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:08.291044 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.290990 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs podName:67e8fbc2-1ffc-42d1-8ea6-307da940eda6 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:10.29096616 +0000 UTC m=+5.946589406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs") pod "network-metrics-daemon-2jq6z" (UID: "67e8fbc2-1ffc-42d1-8ea6-307da940eda6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:08.391461 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.391427 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:08.391648 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.391609 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:08.391716 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.391667 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret podName:80391286-a001-4714-95f7-47d673ca1a6b nodeName:}" failed. No retries permitted until 2026-07-11 23:31:10.391649855 +0000 UTC m=+6.047273092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret") pod "global-pull-secret-syncer-q5tgg" (UID: "80391286-a001-4714-95f7-47d673ca1a6b") : object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:08.495403 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.492697 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:08.495403 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.492874 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 23:31:08.495403 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.492893 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 23:31:08.495403 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.492906 2566 projected.go:196] Error preparing data for projected volume kube-api-access-nhm4x for pod openshift-network-diagnostics/network-check-target-66kvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:08.495403 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.492961 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x podName:2162ffaf-d5ff-4e93-bc07-02a8db8d7caf nodeName:}" failed. No retries permitted until 2026-07-11 23:31:10.49294326 +0000 UTC m=+6.148566505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhm4x" (UniqueName: "kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x") pod "network-check-target-66kvw" (UID: "2162ffaf-d5ff-4e93-bc07-02a8db8d7caf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:08.803363 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.802619 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:08.803363 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.802744 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:08.803363 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.803090 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:08.803363 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.803189 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:08.803363 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.803242 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:08.803363 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:08.803325 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:08.828891 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.828859 2566 generic.go:356] "Generic (PLEG): container finished" podID="6b8a28ec8a6e788a6d3d86a8cc3699c9" containerID="e5711a409749ea6a6cb29fb6a8c60d8c3d8b9ae73f64f6e9535fb497b2c714d9" exitCode=0 Jul 11 23:31:08.829762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.829725 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" event={"ID":"6b8a28ec8a6e788a6d3d86a8cc3699c9","Type":"ContainerDied","Data":"e5711a409749ea6a6cb29fb6a8c60d8c3d8b9ae73f64f6e9535fb497b2c714d9"} Jul 11 23:31:08.851624 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:08.851570 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-251.ec2.internal" podStartSLOduration=3.8515513 podStartE2EDuration="3.8515513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:07.832300534 +0000 UTC m=+3.487923778" watchObservedRunningTime="2026-07-11 23:31:08.8515513 +0000 UTC m=+4.507174554" Jul 11 23:31:09.850816 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:09.850700 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" event={"ID":"6b8a28ec8a6e788a6d3d86a8cc3699c9","Type":"ContainerStarted","Data":"60682bb5daf57dec061194cdc03a24c51ebd14d5aecb3a7decb33a37298df37b"} Jul 11 23:31:09.865603 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:09.865551 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-251.ec2.internal" podStartSLOduration=4.865531539 podStartE2EDuration="4.865531539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:09.864555434 +0000 UTC m=+5.520178715" watchObservedRunningTime="2026-07-11 23:31:09.865531539 +0000 UTC m=+5.521154793" Jul 11 23:31:10.311164 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:10.310923 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:10.311164 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.311115 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:10.311374 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.311173 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs podName:67e8fbc2-1ffc-42d1-8ea6-307da940eda6 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:14.311156108 +0000 UTC m=+9.966779346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs") pod "network-metrics-daemon-2jq6z" (UID: "67e8fbc2-1ffc-42d1-8ea6-307da940eda6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:10.412214 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:10.412107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:10.412383 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.412281 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:10.412383 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.412350 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret podName:80391286-a001-4714-95f7-47d673ca1a6b nodeName:}" failed. No retries permitted until 2026-07-11 23:31:14.412327318 +0000 UTC m=+10.067950563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret") pod "global-pull-secret-syncer-q5tgg" (UID: "80391286-a001-4714-95f7-47d673ca1a6b") : object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:10.512830 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:10.512796 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:10.512995 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.512941 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 23:31:10.512995 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.512958 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 23:31:10.512995 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.512970 2566 projected.go:196] Error preparing data for projected volume kube-api-access-nhm4x for pod openshift-network-diagnostics/network-check-target-66kvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:10.513149 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.513039 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x podName:2162ffaf-d5ff-4e93-bc07-02a8db8d7caf nodeName:}" failed. No retries permitted until 2026-07-11 23:31:14.513020306 +0000 UTC m=+10.168643540 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhm4x" (UniqueName: "kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x") pod "network-check-target-66kvw" (UID: "2162ffaf-d5ff-4e93-bc07-02a8db8d7caf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:10.802086 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:10.801356 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:10.802086 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:10.801383 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:10.802086 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.801486 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:10.802086 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.801607 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:10.802086 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:10.801333 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:10.802086 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:10.801997 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:12.801310 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:12.801262 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:12.801789 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:12.801404 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:12.801789 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:12.801742 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:12.801905 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:12.801842 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:12.801905 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:12.801887 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:12.802003 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:12.801957 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:14.343758 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:14.343720 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:14.344263 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.343862 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:14.344263 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.343923 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs podName:67e8fbc2-1ffc-42d1-8ea6-307da940eda6 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:22.343903092 +0000 UTC m=+17.999526326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs") pod "network-metrics-daemon-2jq6z" (UID: "67e8fbc2-1ffc-42d1-8ea6-307da940eda6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:14.445279 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:14.444837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:14.445279 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.444976 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:14.445279 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.445074 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret podName:80391286-a001-4714-95f7-47d673ca1a6b nodeName:}" failed. No retries permitted until 2026-07-11 23:31:22.445033185 +0000 UTC m=+18.100656439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret") pod "global-pull-secret-syncer-q5tgg" (UID: "80391286-a001-4714-95f7-47d673ca1a6b") : object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:14.545601 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:14.545566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:14.545758 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.545722 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 23:31:14.545758 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.545743 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 23:31:14.545758 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.545755 2566 projected.go:196] Error preparing data for projected volume kube-api-access-nhm4x for pod openshift-network-diagnostics/network-check-target-66kvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:14.545900 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.545818 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x podName:2162ffaf-d5ff-4e93-bc07-02a8db8d7caf nodeName:}" failed. No retries permitted until 2026-07-11 23:31:22.545797801 +0000 UTC m=+18.201421047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhm4x" (UniqueName: "kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x") pod "network-check-target-66kvw" (UID: "2162ffaf-d5ff-4e93-bc07-02a8db8d7caf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:14.802939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:14.802378 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:14.802939 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.802488 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:14.802939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:14.802516 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:14.802939 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.802641 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:14.802939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:14.802677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:14.802939 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:14.802750 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:16.801161 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:16.801113 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:16.801161 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:16.801130 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:16.801664 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:16.801229 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:16.801664 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:16.801322 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:16.801664 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:16.801367 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:16.803285 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:16.801866 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:18.803955 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:18.803927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:18.804410 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:18.803927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:18.804410 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:18.804047 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:18.804410 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:18.803927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:18.804410 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:18.804124 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:18.804410 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:18.804178 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:20.804124 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:20.804083 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:20.804590 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:20.804090 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:20.804590 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:20.804204 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:20.804590 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:20.804090 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:20.804590 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:20.804279 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:20.804590 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:20.804381 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:22.408648 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:22.408616 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:22.409159 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.408776 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:22.409159 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.408841 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs podName:67e8fbc2-1ffc-42d1-8ea6-307da940eda6 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:38.408821582 +0000 UTC m=+34.064444828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs") pod "network-metrics-daemon-2jq6z" (UID: "67e8fbc2-1ffc-42d1-8ea6-307da940eda6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:22.509212 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:22.509183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:22.509343 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.509293 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:22.509402 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.509346 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret podName:80391286-a001-4714-95f7-47d673ca1a6b nodeName:}" failed. No retries permitted until 2026-07-11 23:31:38.509329371 +0000 UTC m=+34.164952601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret") pod "global-pull-secret-syncer-q5tgg" (UID: "80391286-a001-4714-95f7-47d673ca1a6b") : object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:22.610068 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:22.610044 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:22.610230 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.610214 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 23:31:22.610286 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.610237 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 23:31:22.610286 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.610246 2566 projected.go:196] Error preparing data for projected volume kube-api-access-nhm4x for pod openshift-network-diagnostics/network-check-target-66kvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:22.610388 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.610296 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x podName:2162ffaf-d5ff-4e93-bc07-02a8db8d7caf nodeName:}" failed. No retries permitted until 2026-07-11 23:31:38.610282942 +0000 UTC m=+34.265906172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhm4x" (UniqueName: "kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x") pod "network-check-target-66kvw" (UID: "2162ffaf-d5ff-4e93-bc07-02a8db8d7caf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:22.801779 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:22.801703 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:22.801779 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:22.801726 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:22.801998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:22.801703 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:22.801998 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.801813 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:22.801998 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.801935 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:22.802155 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:22.802019 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:24.804487 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:24.802525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:24.804487 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:24.802692 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:24.804487 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:24.803129 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:24.804487 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:24.803241 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:24.804487 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:24.803294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:24.804487 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:24.803385 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:26.802269 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.802058 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:26.802795 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.802081 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:26.802795 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:26.802356 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:26.802795 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.802084 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:26.802795 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:26.802435 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:26.802795 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:26.802543 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:26.883228 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.883143 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-phdpk" event={"ID":"941ffd9d-f5b2-4c5c-837b-e8bf8e26481b","Type":"ContainerStarted","Data":"acacfe97ebc13c359b57dd021bd305cff8a3852526ab9d550845880ac10c309b"} Jul 11 23:31:26.884676 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.884644 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" event={"ID":"a8a9629d-d41e-49b5-b3d1-51140debf751","Type":"ContainerStarted","Data":"b2a7f6ecba20e0925abf575b86d5ce21a2455351da72856b2c4e2c2f669d916e"} Jul 11 23:31:26.887514 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.887484 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:31:26.887898 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.887877 2566 generic.go:356] "Generic (PLEG): container finished" podID="d77fef30-62dd-4edf-b2c9-3d14a4a1a614" containerID="65609810e0699f8d8b445a3265e776cdb5ddeaf74151e7a46549c838a7d74f5c" exitCode=1 Jul 11 23:31:26.887970 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.887936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"11e25f35dc1240fbb1bd1bc661ff81fec2b32032dac63c10e9f25d3d80645945"} Jul 11 23:31:26.888024 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.887985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"b05bbad7f6a9c19f210b9d111fe71fc302c3e78b4f6d7aef7f9ce870f6418b81"} Jul 11 23:31:26.888024 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.888021 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"85abdc52a84f17969dc50eb1f0bc679965dfe97aecd376e073cff5c2b097394c"} Jul 11 23:31:26.888089 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.888057 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"acca01f8062efdbe64281ac4baadd412cf9b18134f35afe57158048b7449b989"} Jul 11 23:31:26.888121 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.888092 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerDied","Data":"65609810e0699f8d8b445a3265e776cdb5ddeaf74151e7a46549c838a7d74f5c"} Jul 11 23:31:26.888204 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.888177 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"4b08172886ca8e3200bc2de4f4dd5f5310a05165fa182f274399c3c4d2d88336"} Jul 11 23:31:26.889313 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.889296 2566 generic.go:356] "Generic (PLEG): container finished" podID="d08914c3-1c8f-4788-9b12-3cb52cb04ba9" containerID="100f0e5e6efc9b5b06ed96736bdae7e7eb939ccf0d5be50d96c9740b064e97e0" exitCode=0 Jul 11 23:31:26.889408 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.889358 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerDied","Data":"100f0e5e6efc9b5b06ed96736bdae7e7eb939ccf0d5be50d96c9740b064e97e0"} Jul 11 23:31:26.890787 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.890751 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qbjd9" event={"ID":"d87e378d-6b17-43a7-8667-e095db0b08b0","Type":"ContainerStarted","Data":"cf07ef4bc93900b12435478446037639d0d6df783c40cdb9ac4804ab27ecfe34"} Jul 11 23:31:26.892201 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.892181 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-f58r4" event={"ID":"67a00afc-3605-4f51-9303-12299c30fd84","Type":"ContainerStarted","Data":"3ff300b69e1b361a10e12ae2579d2cf8178cbd32d5227d162601f360f9d08185"} Jul 11 23:31:26.893671 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.893641 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" event={"ID":"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd","Type":"ContainerStarted","Data":"6543d7dd7678982f749504a242159b2ce8ef11545937e1d78b58b8fb4c27ba58"} Jul 11 23:31:26.894959 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.894941 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lwcsq" event={"ID":"95d3c2ae-d983-46de-a4fd-a417d0059a37","Type":"ContainerStarted","Data":"0393ee14449ee2ff931fd8150f656137f85b1b54e683de9746165d6332a83651"} Jul 11 23:31:26.899355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.899314 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-phdpk" podStartSLOduration=4.371243139 podStartE2EDuration="22.899299282s" totalImagesPullingTime="18.528056143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:26.89897336 +0000 UTC m=+22.554596614" watchObservedRunningTime="2026-07-11 23:31:26.899299282 +0000 UTC m=+22.554922536" Jul 11 23:31:26.937701 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.937667 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k4qzh" podStartSLOduration=3.478076093 podStartE2EDuration="21.93765608s" totalImagesPullingTime="18.459579987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:26.937469614 +0000 UTC m=+22.593092868" watchObservedRunningTime="2026-07-11 23:31:26.93765608 +0000 UTC m=+22.593279356" Jul 11 23:31:26.952803 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.952762 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-f58r4" podStartSLOduration=3.465673699 podStartE2EDuration="21.952747264s" totalImagesPullingTime="18.487073565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:26.952695572 +0000 UTC m=+22.608318850" watchObservedRunningTime="2026-07-11 23:31:26.952747264 +0000 UTC m=+22.608370515" Jul 11 23:31:26.971665 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.971623 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lwcsq" podStartSLOduration=4.513594022 podStartE2EDuration="22.971612872s" totalImagesPullingTime="18.45801885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:26.971516214 +0000 UTC m=+22.627139493" watchObservedRunningTime="2026-07-11 23:31:26.971612872 +0000 UTC m=+22.627236127" Jul 11 23:31:26.989400 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:26.989356 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qbjd9" podStartSLOduration=11.736316511 podStartE2EDuration="21.989340593s" totalImagesPullingTime="10.253024082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:26.988696537 +0000 UTC m=+22.644319789" watchObservedRunningTime="2026-07-11 23:31:26.989340593 +0000 UTC m=+22.644963850" Jul 11 23:31:27.720441 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:27.720421 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 11 23:31:27.741948 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:27.741854 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-11T23:31:27.720436965Z","UUID":"e7f106df-970e-449b-9e54-0471001c536e","Handler":null,"Name":"","Endpoint":""} Jul 11 23:31:27.743047 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:27.743033 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 11 23:31:27.743125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:27.743069 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 11 23:31:27.898868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:27.898780 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2bwwc" event={"ID":"dc515d92-277f-408f-af90-7c8561df5b17","Type":"ContainerStarted","Data":"4359991b70e8538b845cf79891ba414e24d0363caac5fed949eec45af739b735"} Jul 11 23:31:27.900711 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:27.900668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" event={"ID":"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd","Type":"ContainerStarted","Data":"d3932a77c1d3375cd3c85fdc4a6000aeef6dbd38cd0f9a80c7cf2a66c60c0a84"} Jul 11 23:31:27.913193 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:27.913125 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2bwwc" podStartSLOduration=5.426361311 podStartE2EDuration="23.913108225s" totalImagesPullingTime="18.486746914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:27.912854223 +0000 UTC m=+23.568477478" watchObservedRunningTime="2026-07-11 23:31:27.913108225 +0000 UTC m=+23.568731479" Jul 11 23:31:28.802032 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:28.801990 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:28.802304 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:28.802003 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:28.802304 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:28.802136 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:28.802304 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:28.802179 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:28.802304 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:28.802275 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:28.802482 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:28.802373 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:28.964768 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:28.964741 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:28.965561 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:28.965536 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:28.990003 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:28.989981 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:28.990483 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:28.990457 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qbjd9" Jul 11 23:31:29.908912 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:29.908735 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:31:29.909355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:29.909332 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"2ea55a11d9eb6cb362dafea0c45d7ceba10212672128343608b95589dfd693bf"} Jul 11 23:31:29.911802 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:29.911774 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" event={"ID":"957ca13a-7a50-4c5c-9f09-d4bb32adb8cd","Type":"ContainerStarted","Data":"6f0f130ae43d2aa050d169a447a6400613b43d5f715c7a308fe1f4e4552143a4"} Jul 11 23:31:29.928662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:29.928619 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m2qzs" podStartSLOduration=3.942532364 podStartE2EDuration="24.928605092s" totalImagesPullingTime="20.986072728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:29.928047285 +0000 UTC m=+25.583670537" watchObservedRunningTime="2026-07-11 23:31:29.928605092 +0000 UTC m=+25.584228353" Jul 11 23:31:30.801376 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:30.801341 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:30.801376 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:30.801357 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:30.801986 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:30.801468 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:30.801986 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:30.801740 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:30.801986 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:30.801794 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:30.801986 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:30.801920 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:31.918750 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.918592 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:31:31.919399 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.919097 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"9b7326b2612b06112281cfcf5279449366bcb91da3ff4e3f8aa094e819e5078a"} Jul 11 23:31:31.919399 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.919355 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:31.919514 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.919423 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:31.919514 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.919475 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:31.919590 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.919532 2566 scope.go:117] "RemoveContainer" containerID="65609810e0699f8d8b445a3265e776cdb5ddeaf74151e7a46549c838a7d74f5c" Jul 11 23:31:31.921450 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.921421 2566 generic.go:356] "Generic (PLEG): container finished" podID="d08914c3-1c8f-4788-9b12-3cb52cb04ba9" containerID="0c66792aedc10f552cb939252dc37fd2b0221aed052ef4db1d71555a88cdfe79" exitCode=0 Jul 11 23:31:31.921556 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.921455 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerDied","Data":"0c66792aedc10f552cb939252dc37fd2b0221aed052ef4db1d71555a88cdfe79"} Jul 11 23:31:31.934763 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.934741 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:31.934937 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:31.934922 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:31:32.801596 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:32.801564 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:32.801764 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:32.801565 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:32.801764 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:32.801664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:32.801764 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:32.801667 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:32.801764 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:32.801741 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:32.801916 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:32.801818 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:32.927248 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:32.927222 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:31:32.927641 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:32.927594 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" event={"ID":"d77fef30-62dd-4edf-b2c9-3d14a4a1a614","Type":"ContainerStarted","Data":"5ab643d4363e1d277634b4455350735216d54ee4c59efded06d6ba53b190771b"} Jul 11 23:31:32.953483 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:32.953443 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" podStartSLOduration=10.427093224 podStartE2EDuration="28.953428899s" totalImagesPullingTime="18.526335675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:32.952801411 +0000 UTC m=+28.608424664" watchObservedRunningTime="2026-07-11 23:31:32.953428899 +0000 UTC m=+28.609052152" Jul 11 23:31:33.415546 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.415425 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-66kvw"] Jul 11 23:31:33.415546 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.415546 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:33.415764 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:33.415648 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:33.418625 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.418588 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q5tgg"] Jul 11 23:31:33.418738 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.418688 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:33.418801 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:33.418783 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:33.419317 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.419296 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2jq6z"] Jul 11 23:31:33.419424 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.419377 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:33.419495 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:33.419461 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:33.932244 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.932215 2566 generic.go:356] "Generic (PLEG): container finished" podID="d08914c3-1c8f-4788-9b12-3cb52cb04ba9" containerID="1a1f2098a29dade85d2372af1bfce834f374856b4234e7de1b93cf3a2a2e4313" exitCode=0 Jul 11 23:31:33.932649 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:33.932278 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerDied","Data":"1a1f2098a29dade85d2372af1bfce834f374856b4234e7de1b93cf3a2a2e4313"} Jul 11 23:31:34.802760 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:34.802730 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:34.802900 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:34.802831 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:34.802900 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:34.802872 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:34.802900 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:34.802863 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:34.803006 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:34.802966 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:34.803061 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:34.803043 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:35.939558 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:35.939388 2566 generic.go:356] "Generic (PLEG): container finished" podID="d08914c3-1c8f-4788-9b12-3cb52cb04ba9" containerID="0fd72f40c210db3365bfc041c557bc07183ce7c03b96fc5bb88757ced40726ab" exitCode=0 Jul 11 23:31:35.939558 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:35.939473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerDied","Data":"0fd72f40c210db3365bfc041c557bc07183ce7c03b96fc5bb88757ced40726ab"} Jul 11 23:31:36.801351 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:36.801316 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:36.801554 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:36.801316 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:36.801554 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:36.801448 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:36.801554 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:36.801444 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2jq6z" podUID="67e8fbc2-1ffc-42d1-8ea6-307da940eda6" Jul 11 23:31:36.801703 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:36.801570 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q5tgg" podUID="80391286-a001-4714-95f7-47d673ca1a6b" Jul 11 23:31:36.801703 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:36.801676 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-66kvw" podUID="2162ffaf-d5ff-4e93-bc07-02a8db8d7caf" Jul 11 23:31:38.436194 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.436163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:38.436690 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.436336 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:38.436690 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.436410 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs podName:67e8fbc2-1ffc-42d1-8ea6-307da940eda6 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.436388778 +0000 UTC m=+66.092012023 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs") pod "network-metrics-daemon-2jq6z" (UID: "67e8fbc2-1ffc-42d1-8ea6-307da940eda6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 11 23:31:38.537364 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.537319 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:38.537548 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.537479 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:38.537607 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.537558 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret podName:80391286-a001-4714-95f7-47d673ca1a6b nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.537542084 +0000 UTC m=+66.193165330 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret") pod "global-pull-secret-syncer-q5tgg" (UID: "80391286-a001-4714-95f7-47d673ca1a6b") : object "kube-system"/"original-pull-secret" not registered Jul 11 23:31:38.638331 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.638302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:38.638458 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.638446 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 11 23:31:38.638536 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.638469 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 11 23:31:38.638536 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.638481 2566 projected.go:196] Error preparing data for projected volume kube-api-access-nhm4x for pod openshift-network-diagnostics/network-check-target-66kvw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:38.638626 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.638554 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x podName:2162ffaf-d5ff-4e93-bc07-02a8db8d7caf nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.63853639 +0000 UTC m=+66.294159624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhm4x" (UniqueName: "kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x") pod "network-check-target-66kvw" (UID: "2162ffaf-d5ff-4e93-bc07-02a8db8d7caf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 11 23:31:38.708183 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.708128 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-251.ec2.internal" event="NodeReady" Jul 11 23:31:38.708295 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.708262 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 11 23:31:38.738897 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.738871 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75bf7ff465-njvd6"] Jul 11 23:31:38.742883 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.742863 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm"] Jul 11 23:31:38.742995 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.742977 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.745990 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.745971 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 11 23:31:38.746080 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.746001 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 11 23:31:38.746080 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.746032 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h9mt9\"" Jul 11 23:31:38.746165 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.746111 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 11 23:31:38.746452 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.746432 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.751385 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.751349 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 11 23:31:38.752520 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.752482 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xmtxs\"" Jul 11 23:31:38.753040 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.753020 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 11 23:31:38.753136 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.753095 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx"] Jul 11 23:31:38.753136 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.753100 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.753251 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.753227 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.753601 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.753577 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 11 23:31:38.758833 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.757249 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-v692p"] Jul 11 23:31:38.761790 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.761770 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" Jul 11 23:31:38.766606 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.766583 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-4qd7j"] Jul 11 23:31:38.767057 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.767038 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-d72mg\"" Jul 11 23:31:38.769828 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.767585 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.769828 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.768277 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.770789 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.770229 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:38.772314 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.772293 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-rdf9q"] Jul 11 23:31:38.772916 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.772416 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:38.773617 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.773601 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 11 23:31:38.773874 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.773853 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wr2z4\"" Jul 11 23:31:38.774095 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.774081 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 11 23:31:38.774433 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.774260 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.774433 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.774128 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.775262 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.775242 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 11 23:31:38.776142 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776107 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm"] Jul 11 23:31:38.776142 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776141 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75bf7ff465-njvd6"] Jul 11 23:31:38.776283 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776161 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx"] Jul 11 23:31:38.776283 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776201 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-v692p"] Jul 11 23:31:38.776283 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776216 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-4qd7j"] Jul 11 23:31:38.776283 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776235 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg"] Jul 11 23:31:38.776476 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776296 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:38.776606 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776588 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 11 23:31:38.777125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776722 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.777125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.777011 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.777125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.776904 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ptgvv\"" Jul 11 23:31:38.779728 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.779711 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kqrpg"] Jul 11 23:31:38.779947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.779864 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" Jul 11 23:31:38.781724 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.781700 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 11 23:31:38.781939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.781925 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-b74md\"" Jul 11 23:31:38.782717 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.782522 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 11 23:31:38.782816 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.782760 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 11 23:31:38.783195 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.783178 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7tlrm"] Jul 11 23:31:38.784289 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.784267 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:38.786271 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.786069 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.786271 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.786130 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 11 23:31:38.786416 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.786368 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.786952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.786882 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-67fnx\"" Jul 11 23:31:38.787305 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.787090 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m"] Jul 11 23:31:38.787684 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.787635 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:38.790419 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.790373 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh"] Jul 11 23:31:38.790789 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.790770 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:38.792345 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.792328 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8s5q6\"" Jul 11 23:31:38.792967 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.792949 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.794184 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.793697 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.794184 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.793877 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 11 23:31:38.794184 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.793965 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8jgcr\"" Jul 11 23:31:38.794184 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.793999 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 11 23:31:38.794564 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.794546 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.794660 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.794646 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 11 23:31:38.795460 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.795160 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vn6qn\"" Jul 11 23:31:38.796030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.795828 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 11 23:31:38.796030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.795853 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 11 23:31:38.796247 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.796230 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.801968 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.801949 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:38.804596 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.804579 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.804738 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.804725 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 11 23:31:38.806030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.805821 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nfccd\"" Jul 11 23:31:38.806132 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.806117 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.807935 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.807916 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4"] Jul 11 23:31:38.808012 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.807982 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:31:38.808179 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.808107 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:31:38.808253 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.808206 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:31:38.810422 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.810407 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxtg4\"" Jul 11 23:31:38.810530 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.810421 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 11 23:31:38.810530 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.810494 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 11 23:31:38.810752 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.810737 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qb5vv\"" Jul 11 23:31:38.811370 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.811352 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-c7bdd5f5b-nw2lj"] Jul 11 23:31:38.811541 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.811525 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:38.814467 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814448 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-rdf9q"] Jul 11 23:31:38.814567 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814476 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg"] Jul 11 23:31:38.814567 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814493 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kqrpg"] Jul 11 23:31:38.814567 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814533 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m"] Jul 11 23:31:38.814567 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814552 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7tlrm"] Jul 11 23:31:38.814706 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814573 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4"] Jul 11 23:31:38.814706 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814585 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh"] Jul 11 23:31:38.814706 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814602 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c7bdd5f5b-nw2lj"] Jul 11 23:31:38.814706 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.814697 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:38.815385 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.815357 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 11 23:31:38.815595 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.815565 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.815681 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.815658 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 11 23:31:38.815862 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.815841 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.815943 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.815874 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6chxn\"" Jul 11 23:31:38.817238 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.817221 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 11 23:31:38.817480 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.817464 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 11 23:31:38.817585 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.817548 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 11 23:31:38.817692 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.817675 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 11 23:31:38.817777 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.817761 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dkjr6\"" Jul 11 23:31:38.818055 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.817861 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 11 23:31:38.818142 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.817908 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 11 23:31:38.840033 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840012 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22088323-b6b3-4e92-bc92-7b44ad14c8dc-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.840126 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840045 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-bound-sa-token\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.840126 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840061 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8tl\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-kube-api-access-7k8tl\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.840228 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840131 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22088323-b6b3-4e92-bc92-7b44ad14c8dc-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.840228 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.840301 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-image-registry-private-configuration\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.840301 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840288 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-certificates\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.840394 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840344 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jqp\" (UniqueName: \"kubernetes.io/projected/22088323-b6b3-4e92-bc92-7b44ad14c8dc-kube-api-access-28jqp\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.840436 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840395 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-installation-pull-secrets\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.840436 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840427 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-trusted-ca\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.840542 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.840477 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec49533-1d9f-4e5d-9126-2c2d4a256734-ca-trust-extracted\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.905898 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.905870 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:38.940885 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.940864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-bound-sa-token\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.941010 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.940900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8tl\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-kube-api-access-7k8tl\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.941010 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.940932 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:38.941122 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941076 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wj8\" (UniqueName: \"kubernetes.io/projected/ab0354db-3a99-4cab-b4fb-56d690e36c98-kube-api-access-v8wj8\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:38.941172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941131 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.941172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941164 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-tmp\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:38.941260 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941188 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb879140-59b4-4467-adfe-460df17e72be-trusted-ca\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:38.941260 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:38.941260 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:38.941378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941276 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fce62751-789d-4923-8302-a07368e66a11-cert\") pod \"ingress-canary-kqrpg\" (UID: \"fce62751-789d-4923-8302-a07368e66a11\") " pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:38.941378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzrlm\" (UniqueName: \"kubernetes.io/projected/f1db19fb-9d59-4530-b624-e45f5590ff47-kube-api-access-pzrlm\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:38.941378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941346 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:38.941378 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.941365 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 11 23:31:38.941547 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.941382 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf7ff465-njvd6: secret "image-registry-tls" not found Jul 11 23:31:38.941547 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941413 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:38.941547 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-snapshots\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:38.941685 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.941475 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jqp\" (UniqueName: \"kubernetes.io/projected/22088323-b6b3-4e92-bc92-7b44ad14c8dc-kube-api-access-28jqp\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.942254 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:38.942031 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls podName:3ec49533-1d9f-4e5d-9126-2c2d4a256734 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:39.441471139 +0000 UTC m=+35.097094386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls") pod "image-registry-75bf7ff465-njvd6" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734") : secret "image-registry-tls" not found Jul 11 23:31:38.942254 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942112 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-installation-pull-secrets\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.942254 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-trusted-ca\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.942254 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942205 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb879140-59b4-4467-adfe-460df17e72be-config\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:38.942484 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942248 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddd69\" (UniqueName: \"kubernetes.io/projected/6617a3cd-9a8e-4ee7-a0af-d5c767e8b635-kube-api-access-ddd69\") pod \"network-check-source-869dcfcdcd-w2hcg\" (UID: \"6617a3cd-9a8e-4ee7-a0af-d5c767e8b635\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" Jul 11 23:31:38.942484 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942329 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfnx\" (UniqueName: \"kubernetes.io/projected/67d5e183-87ab-406e-a885-243f0e2eff99-kube-api-access-lwfnx\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:38.942484 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec49533-1d9f-4e5d-9126-2c2d4a256734-ca-trust-extracted\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.942484 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942422 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1db19fb-9d59-4530-b624-e45f5590ff47-tmp-dir\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:38.942484 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942468 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-serving-cert\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:38.942721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942527 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb879140-59b4-4467-adfe-460df17e72be-serving-cert\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:38.942721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:38.942721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22088323-b6b3-4e92-bc92-7b44ad14c8dc-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.942721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd4j9\" (UniqueName: \"kubernetes.io/projected/fce62751-789d-4923-8302-a07368e66a11-kube-api-access-cd4j9\") pod \"ingress-canary-kqrpg\" (UID: \"fce62751-789d-4923-8302-a07368e66a11\") " pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:38.942721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-stats-auth\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:38.942959 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-config\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:38.942959 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942776 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w497\" (UniqueName: \"kubernetes.io/projected/fb879140-59b4-4467-adfe-460df17e72be-kube-api-access-2w497\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:38.942959 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-image-registry-private-configuration\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.942959 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942952 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-certificates\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.943118 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.942998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjdf\" (UniqueName: \"kubernetes.io/projected/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-kube-api-access-jvjdf\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:38.943118 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05d45c15-52aa-4126-8ec5-3ab300906ce5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:38.943194 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db19fb-9d59-4530-b624-e45f5590ff47-config-volume\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943704 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/22088323-b6b3-4e92-bc92-7b44ad14c8dc-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943759 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943805 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlc4c\" (UniqueName: \"kubernetes.io/projected/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-kube-api-access-vlc4c\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943813 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-trusted-ca\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943889 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943916 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec49533-1d9f-4e5d-9126-2c2d4a256734-ca-trust-extracted\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.943936 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-kube-api-access-bj2j9\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.944014 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5c4\" (UniqueName: \"kubernetes.io/projected/1edd63df-f7bf-42b1-8384-a5a4d08232ef-kube-api-access-bx5c4\") pod \"volume-data-source-validator-78495597f6-n7hdx\" (UID: \"1edd63df-f7bf-42b1-8384-a5a4d08232ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.944049 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-serving-cert\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.944080 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22088323-b6b3-4e92-bc92-7b44ad14c8dc-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.944111 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-default-certificate\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:38.944207 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.944162 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-certificates\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.947438 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.947302 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/22088323-b6b3-4e92-bc92-7b44ad14c8dc-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.947551 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.947529 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-installation-pull-secrets\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.948430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.947801 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-image-registry-private-configuration\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.949938 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.949910 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-bound-sa-token\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.950389 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.950363 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8tl\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-kube-api-access-7k8tl\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:38.951202 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.951182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jqp\" (UniqueName: \"kubernetes.io/projected/22088323-b6b3-4e92-bc92-7b44ad14c8dc-kube-api-access-28jqp\") pod \"kube-storage-version-migrator-operator-799cb7f958-zvwpm\" (UID: \"22088323-b6b3-4e92-bc92-7b44ad14c8dc\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:38.993016 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:38.992934 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:39.044890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.044864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:39.044990 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.044915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.044990 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.044943 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-snapshots\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.045107 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.044993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb879140-59b4-4467-adfe-460df17e72be-config\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.045107 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddd69\" (UniqueName: \"kubernetes.io/projected/6617a3cd-9a8e-4ee7-a0af-d5c767e8b635-kube-api-access-ddd69\") pod \"network-check-source-869dcfcdcd-w2hcg\" (UID: \"6617a3cd-9a8e-4ee7-a0af-d5c767e8b635\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" Jul 11 23:31:39.045107 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfnx\" (UniqueName: \"kubernetes.io/projected/67d5e183-87ab-406e-a885-243f0e2eff99-kube-api-access-lwfnx\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.045107 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045083 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1db19fb-9d59-4530-b624-e45f5590ff47-tmp-dir\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.045107 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.045029 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-serving-cert\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045134 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb879140-59b4-4467-adfe-460df17e72be-serving-cert\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.045175 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls podName:ab0354db-3a99-4cab-b4fb-56d690e36c98 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:39.545155915 +0000 UTC m=+35.200779169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls") pod "cluster-samples-operator-65db858894-czpjh" (UID: "ab0354db-3a99-4cab-b4fb-56d690e36c98") : secret "samples-operator-tls" not found Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045209 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd4j9\" (UniqueName: \"kubernetes.io/projected/fce62751-789d-4923-8302-a07368e66a11-kube-api-access-cd4j9\") pod \"ingress-canary-kqrpg\" (UID: \"fce62751-789d-4923-8302-a07368e66a11\") " pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.045226 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045232 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-stats-auth\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045264 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-config\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.045279 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls podName:f1db19fb-9d59-4530-b624-e45f5590ff47 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:39.545264309 +0000 UTC m=+35.200887556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls") pod "dns-default-7tlrm" (UID: "f1db19fb-9d59-4530-b624-e45f5590ff47") : secret "dns-default-metrics-tls" not found Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w497\" (UniqueName: \"kubernetes.io/projected/fb879140-59b4-4467-adfe-460df17e72be-kube-api-access-2w497\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjdf\" (UniqueName: \"kubernetes.io/projected/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-kube-api-access-jvjdf\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:39.045378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045368 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05d45c15-52aa-4126-8ec5-3ab300906ce5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db19fb-9d59-4530-b624-e45f5590ff47-config-volume\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045419 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045452 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlc4c\" (UniqueName: \"kubernetes.io/projected/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-kube-api-access-vlc4c\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045482 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045526 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045555 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-kube-api-access-bj2j9\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045600 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5c4\" (UniqueName: \"kubernetes.io/projected/1edd63df-f7bf-42b1-8384-a5a4d08232ef-kube-api-access-bx5c4\") pod \"volume-data-source-validator-78495597f6-n7hdx\" (UID: \"1edd63df-f7bf-42b1-8384-a5a4d08232ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045624 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-serving-cert\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045622 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-snapshots\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-default-certificate\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045689 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb879140-59b4-4467-adfe-460df17e72be-config\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045716 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wj8\" (UniqueName: \"kubernetes.io/projected/ab0354db-3a99-4cab-b4fb-56d690e36c98-kube-api-access-v8wj8\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045750 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-config\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-tmp\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.045947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb879140-59b4-4467-adfe-460df17e72be-trusted-ca\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045842 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045898 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fce62751-789d-4923-8302-a07368e66a11-cert\") pod \"ingress-canary-kqrpg\" (UID: \"fce62751-789d-4923-8302-a07368e66a11\") " pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.045924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzrlm\" (UniqueName: \"kubernetes.io/projected/f1db19fb-9d59-4530-b624-e45f5590ff47-kube-api-access-pzrlm\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.045083 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.046189 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:39.546170857 +0000 UTC m=+35.201794107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : secret "router-metrics-certs-default" not found Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.046317 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-tmp\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.046337 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1db19fb-9d59-4530-b624-e45f5590ff47-tmp-dir\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.046407 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/05d45c15-52aa-4126-8ec5-3ab300906ce5-nginx-conf\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.046438 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.046460 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:39.5464466 +0000 UTC m=+35.202069848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : configmap references non-existent config key: service-ca.crt Jul 11 23:31:39.046703 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.046487 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls podName:ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:39.546475083 +0000 UTC m=+35.202098316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-9qnk4" (UID: "ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80") : secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:39.047259 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.046900 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 11 23:31:39.047259 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.046938 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:39.047259 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.046997 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert podName:05d45c15-52aa-4126-8ec5-3ab300906ce5 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:39.546979242 +0000 UTC m=+35.202602491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rdf9q" (UID: "05d45c15-52aa-4126-8ec5-3ab300906ce5") : secret "networking-console-plugin-cert" not found Jul 11 23:31:39.047259 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.047147 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.047473 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.047449 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1db19fb-9d59-4530-b624-e45f5590ff47-config-volume\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.047820 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.047795 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fb879140-59b4-4467-adfe-460df17e72be-trusted-ca\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.048017 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.047979 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.048783 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.048759 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb879140-59b4-4467-adfe-460df17e72be-serving-cert\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.049242 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.049219 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fce62751-789d-4923-8302-a07368e66a11-cert\") pod \"ingress-canary-kqrpg\" (UID: \"fce62751-789d-4923-8302-a07368e66a11\") " pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:39.049480 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.049451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-stats-auth\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.049984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.049960 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-serving-cert\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.050273 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.050240 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-default-certificate\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.050360 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.050332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-serving-cert\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:39.058282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.058232 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjdf\" (UniqueName: \"kubernetes.io/projected/4f55f05e-4a71-41c8-99fc-a4c481d7d2e7-kube-api-access-jvjdf\") pod \"service-ca-operator-7695fc6c8c-kqh7m\" (UID: \"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:39.059538 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.058957 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlc4c\" (UniqueName: \"kubernetes.io/projected/6057afd9-b8a3-4f14-9f90-af1beeba6e3a-kube-api-access-vlc4c\") pod \"insights-operator-68d9dd94cf-4qd7j\" (UID: \"6057afd9-b8a3-4f14-9f90-af1beeba6e3a\") " pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.059538 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.059467 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfnx\" (UniqueName: \"kubernetes.io/projected/67d5e183-87ab-406e-a885-243f0e2eff99-kube-api-access-lwfnx\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.059675 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.059632 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wj8\" (UniqueName: \"kubernetes.io/projected/ab0354db-3a99-4cab-b4fb-56d690e36c98-kube-api-access-v8wj8\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:39.060110 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.060092 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddd69\" (UniqueName: \"kubernetes.io/projected/6617a3cd-9a8e-4ee7-a0af-d5c767e8b635-kube-api-access-ddd69\") pod \"network-check-source-869dcfcdcd-w2hcg\" (UID: \"6617a3cd-9a8e-4ee7-a0af-d5c767e8b635\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" Jul 11 23:31:39.060822 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.060764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd4j9\" (UniqueName: \"kubernetes.io/projected/fce62751-789d-4923-8302-a07368e66a11-kube-api-access-cd4j9\") pod \"ingress-canary-kqrpg\" (UID: \"fce62751-789d-4923-8302-a07368e66a11\") " pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:39.061097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.061073 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5c4\" (UniqueName: \"kubernetes.io/projected/1edd63df-f7bf-42b1-8384-a5a4d08232ef-kube-api-access-bx5c4\") pod \"volume-data-source-validator-78495597f6-n7hdx\" (UID: \"1edd63df-f7bf-42b1-8384-a5a4d08232ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" Jul 11 23:31:39.061282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.061264 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzrlm\" (UniqueName: \"kubernetes.io/projected/f1db19fb-9d59-4530-b624-e45f5590ff47-kube-api-access-pzrlm\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.061797 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.061780 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w497\" (UniqueName: \"kubernetes.io/projected/fb879140-59b4-4467-adfe-460df17e72be-kube-api-access-2w497\") pod \"console-operator-7866497494-v692p\" (UID: \"fb879140-59b4-4467-adfe-460df17e72be\") " pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.062278 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.062261 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj2j9\" (UniqueName: \"kubernetes.io/projected/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-kube-api-access-bj2j9\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:39.066035 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.066019 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" Jul 11 23:31:39.078842 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.078828 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" Jul 11 23:31:39.091611 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.091589 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:39.101420 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.101402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" Jul 11 23:31:39.114488 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.114466 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" Jul 11 23:31:39.121466 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.121449 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kqrpg" Jul 11 23:31:39.140474 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.140454 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" Jul 11 23:31:39.449637 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.449607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:39.450340 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.449791 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 11 23:31:39.450340 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.449810 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf7ff465-njvd6: secret "image-registry-tls" not found Jul 11 23:31:39.450340 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.449878 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls podName:3ec49533-1d9f-4e5d-9126-2c2d4a256734 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:40.449858081 +0000 UTC m=+36.105481328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls") pod "image-registry-75bf7ff465-njvd6" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734") : secret "image-registry-tls" not found Jul 11 23:31:39.550083 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.550047 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:39.550227 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.550124 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:39.550227 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.550160 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:39.550227 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.550197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.550266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550198 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550291 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:39.550311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550261 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550341 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:40.55032682 +0000 UTC m=+36.205950055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : secret "router-metrics-certs-default" not found Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550370 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls podName:ab0354db-3a99-4cab-b4fb-56d690e36c98 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:40.550349816 +0000 UTC m=+36.205973063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls") pod "cluster-samples-operator-65db858894-czpjh" (UID: "ab0354db-3a99-4cab-b4fb-56d690e36c98") : secret "samples-operator-tls" not found Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550392 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert podName:05d45c15-52aa-4126-8ec5-3ab300906ce5 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:40.550381974 +0000 UTC m=+36.206005213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rdf9q" (UID: "05d45c15-52aa-4126-8ec5-3ab300906ce5") : secret "networking-console-plugin-cert" not found Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550271 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:39.550400 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550398 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 23:31:39.550938 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550427 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls podName:ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:40.55041869 +0000 UTC m=+36.206041937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-9qnk4" (UID: "ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80") : secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:39.550938 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550447 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:40.550433551 +0000 UTC m=+36.206056785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : configmap references non-existent config key: service-ca.crt Jul 11 23:31:39.550938 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:39.550466 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls podName:f1db19fb-9d59-4530-b624-e45f5590ff47 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:40.550456618 +0000 UTC m=+36.206079855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls") pod "dns-default-7tlrm" (UID: "f1db19fb-9d59-4530-b624-e45f5590ff47") : secret "dns-default-metrics-tls" not found Jul 11 23:31:40.462111 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.459711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:40.462111 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.460322 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 11 23:31:40.462111 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.460341 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf7ff465-njvd6: secret "image-registry-tls" not found Jul 11 23:31:40.462111 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.460405 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls podName:3ec49533-1d9f-4e5d-9126-2c2d4a256734 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:42.460381305 +0000 UTC m=+38.116004559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls") pod "image-registry-75bf7ff465-njvd6" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734") : secret "image-registry-tls" not found Jul 11 23:31:40.534805 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.534766 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx"] Jul 11 23:31:40.539481 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:40.539452 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edd63df_f7bf_42b1_8384_a5a4d08232ef.slice/crio-4b4c0ab90837d8333e325ef348a05375c1704f77166c27e7d82b81f15fa91817 WatchSource:0}: Error finding container 4b4c0ab90837d8333e325ef348a05375c1704f77166c27e7d82b81f15fa91817: Status 404 returned error can't find the container with id 4b4c0ab90837d8333e325ef348a05375c1704f77166c27e7d82b81f15fa91817 Jul 11 23:31:40.547445 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.547421 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg"] Jul 11 23:31:40.549714 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.549693 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-4qd7j"] Jul 11 23:31:40.550165 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:40.550143 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6617a3cd_9a8e_4ee7_a0af_d5c767e8b635.slice/crio-886fcb1d0f11ca03161a07ec7d02a541d4e3b9e0f1056be81c767d2e79c15ea4 WatchSource:0}: Error finding container 886fcb1d0f11ca03161a07ec7d02a541d4e3b9e0f1056be81c767d2e79c15ea4: Status 404 returned error can't find the container with id 886fcb1d0f11ca03161a07ec7d02a541d4e3b9e0f1056be81c767d2e79c15ea4 Jul 11 23:31:40.553691 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.553674 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-v692p"] Jul 11 23:31:40.554099 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:40.554076 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6057afd9_b8a3_4f14_9f90_af1beeba6e3a.slice/crio-723c28a1fe37859f12b07def0e48b42a804192019ee607152179f73cf0502987 WatchSource:0}: Error finding container 723c28a1fe37859f12b07def0e48b42a804192019ee607152179f73cf0502987: Status 404 returned error can't find the container with id 723c28a1fe37859f12b07def0e48b42a804192019ee607152179f73cf0502987 Jul 11 23:31:40.554485 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.554465 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kqrpg"] Jul 11 23:31:40.559071 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:40.559049 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb879140_59b4_4467_adfe_460df17e72be.slice/crio-58925bc7f8d353339027c16d5e8b4561f69e6e0f15088853835dc9509b48f2d9 WatchSource:0}: Error finding container 58925bc7f8d353339027c16d5e8b4561f69e6e0f15088853835dc9509b48f2d9: Status 404 returned error can't find the container with id 58925bc7f8d353339027c16d5e8b4561f69e6e0f15088853835dc9509b48f2d9 Jul 11 23:31:40.560703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.560633 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:40.560703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.560686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:40.560804 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.560754 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:40.560804 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.560786 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 11 23:31:40.560927 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.560803 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:40.560927 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.560845 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls podName:ab0354db-3a99-4cab-b4fb-56d690e36c98 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:42.560826033 +0000 UTC m=+38.216449263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls") pod "cluster-samples-operator-65db858894-czpjh" (UID: "ab0354db-3a99-4cab-b4fb-56d690e36c98") : secret "samples-operator-tls" not found Jul 11 23:31:40.560927 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.560871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:40.560927 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.560916 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:42.560903824 +0000 UTC m=+38.216527062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : configmap references non-existent config key: service-ca.crt Jul 11 23:31:40.561077 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.560960 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:40.561077 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.560963 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 11 23:31:40.561077 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.561018 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 23:31:40.561077 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.561042 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:42.561032016 +0000 UTC m=+38.216655252 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : secret "router-metrics-certs-default" not found Jul 11 23:31:40.561077 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.561057 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls podName:f1db19fb-9d59-4530-b624-e45f5590ff47 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:42.561048468 +0000 UTC m=+38.216671702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls") pod "dns-default-7tlrm" (UID: "f1db19fb-9d59-4530-b624-e45f5590ff47") : secret "dns-default-metrics-tls" not found Jul 11 23:31:40.561281 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:40.561082 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce62751_789d_4923_8302_a07368e66a11.slice/crio-f6973786f4c84275c13cad92f728955f239765b12d2933a8dea636a01c8957e2 WatchSource:0}: Error finding container f6973786f4c84275c13cad92f728955f239765b12d2933a8dea636a01c8957e2: Status 404 returned error can't find the container with id f6973786f4c84275c13cad92f728955f239765b12d2933a8dea636a01c8957e2 Jul 11 23:31:40.561281 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.561129 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 11 23:31:40.561281 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.561139 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:40.561281 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.561167 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert podName:05d45c15-52aa-4126-8ec5-3ab300906ce5 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:42.56115533 +0000 UTC m=+38.216778575 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rdf9q" (UID: "05d45c15-52aa-4126-8ec5-3ab300906ce5") : secret "networking-console-plugin-cert" not found Jul 11 23:31:40.561281 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:40.561195 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls podName:ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:42.56117437 +0000 UTC m=+38.216797605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-9qnk4" (UID: "ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80") : secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:40.567137 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.567109 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m"] Jul 11 23:31:40.568264 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.568246 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm"] Jul 11 23:31:40.570823 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:40.570801 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22088323_b6b3_4e92_bc92_7b44ad14c8dc.slice/crio-00f59317ecb47097637faf3337d21322275b9254fa25ed2e12fecd0579878b09 WatchSource:0}: Error finding container 00f59317ecb47097637faf3337d21322275b9254fa25ed2e12fecd0579878b09: Status 404 returned error can't find the container with id 00f59317ecb47097637faf3337d21322275b9254fa25ed2e12fecd0579878b09 Jul 11 23:31:40.571876 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:31:40.571856 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f55f05e_4a71_41c8_99fc_a4c481d7d2e7.slice/crio-f20866714eb65acf4a6c53e38bfeef1eef38df4817deaed88bd71e248d137620 WatchSource:0}: Error finding container f20866714eb65acf4a6c53e38bfeef1eef38df4817deaed88bd71e248d137620: Status 404 returned error can't find the container with id f20866714eb65acf4a6c53e38bfeef1eef38df4817deaed88bd71e248d137620 Jul 11 23:31:40.955408 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.955374 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-v692p" event={"ID":"fb879140-59b4-4467-adfe-460df17e72be","Type":"ContainerStarted","Data":"58925bc7f8d353339027c16d5e8b4561f69e6e0f15088853835dc9509b48f2d9"} Jul 11 23:31:40.956314 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.956294 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" event={"ID":"22088323-b6b3-4e92-bc92-7b44ad14c8dc","Type":"ContainerStarted","Data":"00f59317ecb47097637faf3337d21322275b9254fa25ed2e12fecd0579878b09"} Jul 11 23:31:40.957239 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.957217 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" event={"ID":"6617a3cd-9a8e-4ee7-a0af-d5c767e8b635","Type":"ContainerStarted","Data":"886fcb1d0f11ca03161a07ec7d02a541d4e3b9e0f1056be81c767d2e79c15ea4"} Jul 11 23:31:40.959686 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.959661 2566 generic.go:356] "Generic (PLEG): container finished" podID="d08914c3-1c8f-4788-9b12-3cb52cb04ba9" containerID="c6858fc5bde2c3fde59143aa9801f2b851d102ff2fff8d2eb1f36d99292081d8" exitCode=0 Jul 11 23:31:40.959787 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.959723 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerDied","Data":"c6858fc5bde2c3fde59143aa9801f2b851d102ff2fff8d2eb1f36d99292081d8"} Jul 11 23:31:40.960782 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.960762 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" event={"ID":"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7","Type":"ContainerStarted","Data":"f20866714eb65acf4a6c53e38bfeef1eef38df4817deaed88bd71e248d137620"} Jul 11 23:31:40.961771 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.961749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kqrpg" event={"ID":"fce62751-789d-4923-8302-a07368e66a11","Type":"ContainerStarted","Data":"f6973786f4c84275c13cad92f728955f239765b12d2933a8dea636a01c8957e2"} Jul 11 23:31:40.962806 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.962785 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" event={"ID":"6057afd9-b8a3-4f14-9f90-af1beeba6e3a","Type":"ContainerStarted","Data":"723c28a1fe37859f12b07def0e48b42a804192019ee607152179f73cf0502987"} Jul 11 23:31:40.963840 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:40.963821 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" event={"ID":"1edd63df-f7bf-42b1-8384-a5a4d08232ef","Type":"ContainerStarted","Data":"4b4c0ab90837d8333e325ef348a05375c1704f77166c27e7d82b81f15fa91817"} Jul 11 23:31:41.980676 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:41.979710 2566 generic.go:356] "Generic (PLEG): container finished" podID="d08914c3-1c8f-4788-9b12-3cb52cb04ba9" containerID="bd3aeac105ef3e0c76effd036852e211f37958bd570f07f93dcf937913baa895" exitCode=0 Jul 11 23:31:41.980676 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:41.979762 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerDied","Data":"bd3aeac105ef3e0c76effd036852e211f37958bd570f07f93dcf937913baa895"} Jul 11 23:31:42.477337 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.477299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:42.477495 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.477470 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 11 23:31:42.477495 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.477493 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf7ff465-njvd6: secret "image-registry-tls" not found Jul 11 23:31:42.477628 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.477582 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls podName:3ec49533-1d9f-4e5d-9126-2c2d4a256734 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:46.477559365 +0000 UTC m=+42.133182620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls") pod "image-registry-75bf7ff465-njvd6" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734") : secret "image-registry-tls" not found Jul 11 23:31:42.578189 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.577991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:42.578304 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.578237 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:42.578304 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.578294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:42.578423 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.578364 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:42.578423 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.578395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:42.578536 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.578442 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:42.578536 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578137 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 23:31:42.578644 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578561 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls podName:f1db19fb-9d59-4530-b624-e45f5590ff47 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:46.578534553 +0000 UTC m=+42.234157807 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls") pod "dns-default-7tlrm" (UID: "f1db19fb-9d59-4530-b624-e45f5590ff47") : secret "dns-default-metrics-tls" not found Jul 11 23:31:42.578644 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578589 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:46.578572921 +0000 UTC m=+42.234196162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : configmap references non-existent config key: service-ca.crt Jul 11 23:31:42.578644 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578594 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 11 23:31:42.578801 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578643 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 11 23:31:42.578801 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578652 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls podName:ab0354db-3a99-4cab-b4fb-56d690e36c98 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:46.578636165 +0000 UTC m=+42.234259420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls") pod "cluster-samples-operator-65db858894-czpjh" (UID: "ab0354db-3a99-4cab-b4fb-56d690e36c98") : secret "samples-operator-tls" not found Jul 11 23:31:42.578801 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578699 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 11 23:31:42.578801 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578744 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert podName:05d45c15-52aa-4126-8ec5-3ab300906ce5 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:46.578729925 +0000 UTC m=+42.234353173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rdf9q" (UID: "05d45c15-52aa-4126-8ec5-3ab300906ce5") : secret "networking-console-plugin-cert" not found Jul 11 23:31:42.578801 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578773 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:42.578801 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578789 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:46.578776706 +0000 UTC m=+42.234399952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : secret "router-metrics-certs-default" not found Jul 11 23:31:42.579134 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:42.578835 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls podName:ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:46.578828151 +0000 UTC m=+42.234451382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-9qnk4" (UID: "ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80") : secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:42.776641 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.776552 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:42.878298 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.878209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:42.988903 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:42.988872 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" event={"ID":"d08914c3-1c8f-4788-9b12-3cb52cb04ba9","Type":"ContainerStarted","Data":"05e2ffa68c02a156c181db928e93f3b72a0f81b48c855a1ba605b4aae8d9a0e8"} Jul 11 23:31:43.019871 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:43.019766 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gzbk4" podStartSLOduration=8.305806481 podStartE2EDuration="38.019747229s" totalImagesPullingTime="29.713940748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:43.017896106 +0000 UTC m=+38.673519389" watchObservedRunningTime="2026-07-11 23:31:43.019747229 +0000 UTC m=+38.675370483" Jul 11 23:31:43.112267 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:43.112178 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:43.182685 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:43.182642 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:43.311640 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:43.311605 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:43.362047 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:43.362016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:43.478081 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:43.477558 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:43.995594 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:43.995522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" event={"ID":"1edd63df-f7bf-42b1-8384-a5a4d08232ef","Type":"ContainerStarted","Data":"c0a0fb4c6948b618279ae66c98635f516d068dc5901444471b376b712ea83abd"} Jul 11 23:31:44.011973 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:44.011918 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-n7hdx" podStartSLOduration=4.697275909 podStartE2EDuration="7.011897458s" totalImagesPullingTime="2.314621549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:44.010316696 +0000 UTC m=+39.665939951" watchObservedRunningTime="2026-07-11 23:31:44.011897458 +0000 UTC m=+39.667520712" Jul 11 23:31:44.091490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:44.091462 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:44.191938 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:44.191903 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:44.292022 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:44.291922 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:44.481234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:44.481207 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:44.538186 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:44.538148 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:46.513226 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.513189 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:46.513651 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.513317 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 11 23:31:46.513651 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.513330 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf7ff465-njvd6: secret "image-registry-tls" not found Jul 11 23:31:46.513651 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.513387 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls podName:3ec49533-1d9f-4e5d-9126-2c2d4a256734 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:54.513368878 +0000 UTC m=+50.168992111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls") pod "image-registry-75bf7ff465-njvd6" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734") : secret "image-registry-tls" not found Jul 11 23:31:46.613845 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.613808 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:46.614022 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.613897 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:46.614022 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.613945 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:46.614022 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.613966 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 23:31:46.614022 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.614011 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614041 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls podName:f1db19fb-9d59-4530-b624-e45f5590ff47 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:54.614017966 +0000 UTC m=+50.269641217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls") pod "dns-default-7tlrm" (UID: "f1db19fb-9d59-4530-b624-e45f5590ff47") : secret "dns-default-metrics-tls" not found Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.614085 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614100 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614114 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:54.614095767 +0000 UTC m=+50.269718999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : configmap references non-existent config key: service-ca.crt Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614158 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert podName:05d45c15-52aa-4126-8ec5-3ab300906ce5 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:54.614146061 +0000 UTC m=+50.269769292 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rdf9q" (UID: "05d45c15-52aa-4126-8ec5-3ab300906ce5") : secret "networking-console-plugin-cert" not found Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614165 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.614188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614194 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls podName:ab0354db-3a99-4cab-b4fb-56d690e36c98 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:54.614184324 +0000 UTC m=+50.269807555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls") pod "cluster-samples-operator-65db858894-czpjh" (UID: "ab0354db-3a99-4cab-b4fb-56d690e36c98") : secret "samples-operator-tls" not found Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614203 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:46.614247 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614250 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls podName:ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:54.614239275 +0000 UTC m=+50.269862506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-9qnk4" (UID: "ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80") : secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:46.614668 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614262 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 11 23:31:46.614668 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:46.614290 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:31:54.614279196 +0000 UTC m=+50.269902428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : secret "router-metrics-certs-default" not found Jul 11 23:31:46.852533 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.852431 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:46.900834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:46.900800 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:49.016062 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.015974 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" event={"ID":"22088323-b6b3-4e92-bc92-7b44ad14c8dc","Type":"ContainerStarted","Data":"98fd543da5e27360f4861992fce6a0bd87868cb2ed2c8274afdeb35c8439cbe3"} Jul 11 23:31:49.018703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.018670 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" event={"ID":"6617a3cd-9a8e-4ee7-a0af-d5c767e8b635","Type":"ContainerStarted","Data":"afd669b9a692667f0f9fd651897f490a64ec84aed54ef76e218437628c5d75f9"} Jul 11 23:31:49.020238 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.020212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" event={"ID":"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7","Type":"ContainerStarted","Data":"bf682f916cb232264cf7d3841aaec94781c942b77fe44fc41f02ade5c4fe6df6"} Jul 11 23:31:49.022009 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.021972 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kqrpg" event={"ID":"fce62751-789d-4923-8302-a07368e66a11","Type":"ContainerStarted","Data":"13dbcc6763b737d109029ba93f2f991c32cc39521f0fa8471997fef5c144b403"} Jul 11 23:31:49.023906 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.023879 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" event={"ID":"6057afd9-b8a3-4f14-9f90-af1beeba6e3a","Type":"ContainerStarted","Data":"55bc1204def07c8182af40707f9e0099e1714c6b6d4af263b90c676fb33681c1"} Jul 11 23:31:49.025762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.025742 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/0.log" Jul 11 23:31:49.025850 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.025782 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb879140-59b4-4467-adfe-460df17e72be" containerID="66568d6632f78ab4831e4701e7fbd77ff26f0454c59eea666e23b56f579ef284" exitCode=255 Jul 11 23:31:49.025850 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.025815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-v692p" event={"ID":"fb879140-59b4-4467-adfe-460df17e72be","Type":"ContainerDied","Data":"66568d6632f78ab4831e4701e7fbd77ff26f0454c59eea666e23b56f579ef284"} Jul 11 23:31:49.026024 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.026008 2566 scope.go:117] "RemoveContainer" containerID="66568d6632f78ab4831e4701e7fbd77ff26f0454c59eea666e23b56f579ef284" Jul 11 23:31:49.032560 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.032496 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" podStartSLOduration=4.19822901 podStartE2EDuration="12.032481811s" totalImagesPullingTime="7.834252801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:49.031825501 +0000 UTC m=+44.687448758" watchObservedRunningTime="2026-07-11 23:31:49.032481811 +0000 UTC m=+44.688105065" Jul 11 23:31:49.064059 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.064015 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" podStartSLOduration=4.213682617 podStartE2EDuration="12.064002748s" totalImagesPullingTime="7.850320131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:49.06269624 +0000 UTC m=+44.718319519" watchObservedRunningTime="2026-07-11 23:31:49.064002748 +0000 UTC m=+44.719626001" Jul 11 23:31:49.084584 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.084242 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-w2hcg" podStartSLOduration=4.220109344 podStartE2EDuration="12.08422446s" totalImagesPullingTime="7.864115116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:49.08320776 +0000 UTC m=+44.738831014" watchObservedRunningTime="2026-07-11 23:31:49.08422446 +0000 UTC m=+44.739847710" Jul 11 23:31:49.092424 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.092374 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:49.092556 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.092438 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:49.100797 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.100757 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kqrpg" podStartSLOduration=3.257347604 podStartE2EDuration="11.100740312s" totalImagesPullingTime="7.843392708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:49.100023958 +0000 UTC m=+44.755647208" watchObservedRunningTime="2026-07-11 23:31:49.100740312 +0000 UTC m=+44.756363569" Jul 11 23:31:49.354894 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.354864 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:49.455610 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.455529 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:49.712181 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.712147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:49.807594 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.807559 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:49.866272 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.866239 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:49.912179 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:49.912145 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.030695 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.030634 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:31:50.031035 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.031023 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/0.log" Jul 11 23:31:50.031076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.031056 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb879140-59b4-4467-adfe-460df17e72be" containerID="d7143309eccc8e4f8446a3763593610123d3545d16c91bf5232d8a045457f9f5" exitCode=255 Jul 11 23:31:50.031197 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.031165 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-v692p" event={"ID":"fb879140-59b4-4467-adfe-460df17e72be","Type":"ContainerDied","Data":"d7143309eccc8e4f8446a3763593610123d3545d16c91bf5232d8a045457f9f5"} Jul 11 23:31:50.031318 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.031234 2566 scope.go:117] "RemoveContainer" containerID="66568d6632f78ab4831e4701e7fbd77ff26f0454c59eea666e23b56f579ef284" Jul 11 23:31:50.031393 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.031376 2566 scope.go:117] "RemoveContainer" containerID="d7143309eccc8e4f8446a3763593610123d3545d16c91bf5232d8a045457f9f5" Jul 11 23:31:50.031635 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.031581 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-v692p_openshift-console-operator(fb879140-59b4-4467-adfe-460df17e72be)\"" pod="openshift-console-operator/console-operator-7866497494-v692p" podUID="fb879140-59b4-4467-adfe-460df17e72be" Jul 11 23:31:50.047914 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.047865 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" podStartSLOduration=5.215096577 podStartE2EDuration="13.047847815s" totalImagesPullingTime="7.832751238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:31:49.11854906 +0000 UTC m=+44.774172332" watchObservedRunningTime="2026-07-11 23:31:50.047847815 +0000 UTC m=+45.703471070" Jul 11 23:31:50.133806 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.133776 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.185724 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.185699 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.267099 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.267071 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2l9dx"] Jul 11 23:31:50.295911 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.295849 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2l9dx"] Jul 11 23:31:50.296015 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.295975 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.298384 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.298360 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 11 23:31:50.298524 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.298464 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k95xv\"" Jul 11 23:31:50.298587 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.298538 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 11 23:31:50.317001 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.316974 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.350116 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.350091 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7c0b6ae-142a-40a8-ab9b-c56878988bed-crio-socket\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.350214 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.350160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.350214 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.350203 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2sbg\" (UniqueName: \"kubernetes.io/projected/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-api-access-p2sbg\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.350284 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.350229 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c0b6ae-142a-40a8-ab9b-c56878988bed-data-volume\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.350333 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.350285 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.375105 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.375055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.451334 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.451313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.451445 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.451363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sbg\" (UniqueName: \"kubernetes.io/projected/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-api-access-p2sbg\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.451445 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.451430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c0b6ae-142a-40a8-ab9b-c56878988bed-data-volume\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.451598 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.451468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.451598 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.451576 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7c0b6ae-142a-40a8-ab9b-c56878988bed-crio-socket\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.451685 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.451647 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 23:31:50.451685 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.451669 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 23:31:50.451765 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.451711 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:50.95169352 +0000 UTC m=+46.607316764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : secret "insights-runtime-extractor-tls" not found Jul 11 23:31:50.451765 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.451726 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:50.951718425 +0000 UTC m=+46.607341663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : configmap "kube-rbac-proxy" not found Jul 11 23:31:50.451765 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.451741 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7c0b6ae-142a-40a8-ab9b-c56878988bed-data-volume\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.451885 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.451773 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7c0b6ae-142a-40a8-ab9b-c56878988bed-crio-socket\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.463426 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.463409 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2sbg\" (UniqueName: \"kubernetes.io/projected/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-api-access-p2sbg\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.571234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.571163 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.836441 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.836362 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.939666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.939638 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:50.956668 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.956646 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.956751 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:50.956708 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:50.956800 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.956786 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 23:31:50.956839 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.956810 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 23:31:50.956872 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.956841 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:51.956827288 +0000 UTC m=+47.612450519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : secret "insights-runtime-extractor-tls" not found Jul 11 23:31:50.956872 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:50.956859 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:51.956847674 +0000 UTC m=+47.612470905 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : configmap "kube-rbac-proxy" not found Jul 11 23:31:51.034920 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:51.034905 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:31:51.035214 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:51.035199 2566 scope.go:117] "RemoveContainer" containerID="d7143309eccc8e4f8446a3763593610123d3545d16c91bf5232d8a045457f9f5" Jul 11 23:31:51.035367 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:51.035348 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-v692p_openshift-console-operator(fb879140-59b4-4467-adfe-460df17e72be)\"" pod="openshift-console-operator/console-operator-7866497494-v692p" podUID="fb879140-59b4-4467-adfe-460df17e72be" Jul 11 23:31:51.481298 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:51.481270 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:51.526564 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:51.526540 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:51.964571 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:51.964546 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:51.964750 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:51.964638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:51.964750 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:51.964686 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 23:31:51.964750 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:51.964726 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 23:31:51.964750 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:51.964742 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:53.964727932 +0000 UTC m=+49.620351162 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : secret "insights-runtime-extractor-tls" not found Jul 11 23:31:51.964957 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:51.964786 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:53.964770727 +0000 UTC m=+49.620393959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : configmap "kube-rbac-proxy" not found Jul 11 23:31:52.306772 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:52.306701 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:52.368204 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:52.368175 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:52.476476 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:52.476452 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:52.667858 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:52.667839 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f58r4_67a00afc-3605-4f51-9303-12299c30fd84/dns-node-resolver/0.log" Jul 11 23:31:53.273468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:53.273440 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:53.469086 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:53.469063 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lwcsq_95d3c2ae-d983-46de-a4fd-a417d0059a37/node-ca/0.log" Jul 11 23:31:53.980734 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:53.980698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:53.980926 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:53.980770 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:53.980926 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:53.980841 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 23:31:53.980926 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:53.980906 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 11 23:31:53.981082 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:53.980913 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:57.980897128 +0000 UTC m=+53.636520380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : secret "insights-runtime-extractor-tls" not found Jul 11 23:31:53.981082 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:53.980979 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:31:57.980960801 +0000 UTC m=+53.636584044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : configmap "kube-rbac-proxy" not found Jul 11 23:31:54.469495 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.469465 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kqrpg_fce62751-789d-4923-8302-a07368e66a11/serve-healthcheck-canary/0.log" Jul 11 23:31:54.585198 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.585168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:31:54.585341 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.585327 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 11 23:31:54.585379 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.585342 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75bf7ff465-njvd6: secret "image-registry-tls" not found Jul 11 23:31:54.585412 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.585392 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls podName:3ec49533-1d9f-4e5d-9126-2c2d4a256734 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.585376694 +0000 UTC m=+66.240999933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls") pod "image-registry-75bf7ff465-njvd6" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734") : secret "image-registry-tls" not found Jul 11 23:31:54.686004 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.685974 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:31:54.686134 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.686011 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:31:54.686134 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.686043 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:54.686134 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686127 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:54.686228 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.686143 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:31:54.686228 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686180 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls podName:ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.686164299 +0000 UTC m=+66.341787549 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-9qnk4" (UID: "ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80") : secret "cluster-monitoring-operator-tls" not found Jul 11 23:31:54.686228 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686199 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 11 23:31:54.686228 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686224 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 11 23:31:54.686353 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.686231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:31:54.686353 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686239 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls podName:ab0354db-3a99-4cab-b4fb-56d690e36c98 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.686226857 +0000 UTC m=+66.341850107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls") pod "cluster-samples-operator-65db858894-czpjh" (UID: "ab0354db-3a99-4cab-b4fb-56d690e36c98") : secret "samples-operator-tls" not found Jul 11 23:31:54.686353 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686267 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls podName:f1db19fb-9d59-4530-b624-e45f5590ff47 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.686252894 +0000 UTC m=+66.341876142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls") pod "dns-default-7tlrm" (UID: "f1db19fb-9d59-4530-b624-e45f5590ff47") : secret "dns-default-metrics-tls" not found Jul 11 23:31:54.686353 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686272 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 11 23:31:54.686353 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:54.686299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:31:54.686353 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686308 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.686298127 +0000 UTC m=+66.341921384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : secret "router-metrics-certs-default" not found Jul 11 23:31:54.686353 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686332 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle podName:67d5e183-87ab-406e-a885-243f0e2eff99 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.686321379 +0000 UTC m=+66.341944610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle") pod "router-default-c7bdd5f5b-nw2lj" (UID: "67d5e183-87ab-406e-a885-243f0e2eff99") : configmap references non-existent config key: service-ca.crt Jul 11 23:31:54.686593 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686360 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 11 23:31:54.686593 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:54.686392 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert podName:05d45c15-52aa-4126-8ec5-3ab300906ce5 nodeName:}" failed. No retries permitted until 2026-07-11 23:32:10.686385237 +0000 UTC m=+66.342008467 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rdf9q" (UID: "05d45c15-52aa-4126-8ec5-3ab300906ce5") : secret "networking-console-plugin-cert" not found Jul 11 23:31:55.070188 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:55.070146 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zvwpm_22088323-b6b3-4e92-bc92-7b44ad14c8dc/kube-storage-version-migrator-operator/0.log" Jul 11 23:31:55.555717 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:55.555660 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:55.822158 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:55.822090 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:56.591762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:56.591726 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:56.692172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:56.692148 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:56.792271 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:56.792247 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:56.895019 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:56.894997 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:57.016700 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:57.016673 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:57.494425 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:57.494389 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:57.680571 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:57.680542 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:58.011844 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:58.011813 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:58.012006 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:58.011878 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:58.012006 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:58.011973 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 11 23:31:58.012100 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:58.012036 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls podName:d7c0b6ae-142a-40a8-ab9b-c56878988bed nodeName:}" failed. No retries permitted until 2026-07-11 23:32:06.01202006 +0000 UTC m=+61.667643295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2l9dx" (UID: "d7c0b6ae-142a-40a8-ab9b-c56878988bed") : secret "insights-runtime-extractor-tls" not found Jul 11 23:31:58.012380 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:58.012362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7c0b6ae-142a-40a8-ab9b-c56878988bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:31:58.681938 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:58.681897 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:58.879227 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:58.879190 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:59.085106 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:59.085028 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:59.091789 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:59.091766 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:59.091844 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:59.091819 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:31:59.092139 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:59.092126 2566 scope.go:117] "RemoveContainer" containerID="d7143309eccc8e4f8446a3763593610123d3545d16c91bf5232d8a045457f9f5" Jul 11 23:31:59.092304 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:31:59.092284 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-v692p_openshift-console-operator(fb879140-59b4-4467-adfe-460df17e72be)\"" pod="openshift-console-operator/console-operator-7866497494-v692p" podUID="fb879140-59b4-4467-adfe-460df17e72be" Jul 11 23:31:59.480565 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:59.480534 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:31:59.875841 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:31:59.875763 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:00.077743 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:00.077705 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:00.289330 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:00.289294 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:00.337708 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:00.337675 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:00.481643 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:00.481615 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:00.685641 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:00.685618 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:01.082301 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:01.082228 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:01.127478 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:01.127450 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:01.877449 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:01.877396 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:02.278594 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:02.278560 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:02.379287 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:02.379254 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:02.433366 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:02.433338 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:02.480452 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:02.480423 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:02.679761 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:02.679729 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:03.078280 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:03.078208 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:03.477345 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:03.477288 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:03.942587 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:03.942548 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tvxms" Jul 11 23:32:04.282205 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:04.282166 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:04.486576 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:04.486471 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:04.882289 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:04.882195 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:05.080944 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:05.080914 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:05.481413 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:05.481379 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:05.942166 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:05.942141 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:05.989281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:05.989233 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:06.080166 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.080137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:32:06.082518 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.082484 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7c0b6ae-142a-40a8-ab9b-c56878988bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2l9dx\" (UID: \"d7c0b6ae-142a-40a8-ab9b-c56878988bed\") " pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:32:06.083533 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.083512 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:06.207686 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.207631 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k95xv\"" Jul 11 23:32:06.215667 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.215648 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2l9dx" Jul 11 23:32:06.337322 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.337283 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2l9dx"] Jul 11 23:32:06.339591 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:06.339566 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c0b6ae_142a_40a8_ab9b_c56878988bed.slice/crio-9be1d0077ff8b2aa3dfbab408ff1def4445f8ec7c6c1bbfedf0bf08755703dec WatchSource:0}: Error finding container 9be1d0077ff8b2aa3dfbab408ff1def4445f8ec7c6c1bbfedf0bf08755703dec: Status 404 returned error can't find the container with id 9be1d0077ff8b2aa3dfbab408ff1def4445f8ec7c6c1bbfedf0bf08755703dec Jul 11 23:32:06.482886 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.482830 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:06.882946 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:06.882863 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:07.078033 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:07.077952 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:07.087575 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:07.087542 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2l9dx" event={"ID":"d7c0b6ae-142a-40a8-ab9b-c56878988bed","Type":"ContainerStarted","Data":"9be1d0077ff8b2aa3dfbab408ff1def4445f8ec7c6c1bbfedf0bf08755703dec"} Jul 11 23:32:07.277767 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:07.277597 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:09.802351 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:09.802318 2566 scope.go:117] "RemoveContainer" containerID="d7143309eccc8e4f8446a3763593610123d3545d16c91bf5232d8a045457f9f5" Jul 11 23:32:10.100224 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.100163 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:32:10.100330 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.100281 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-v692p" event={"ID":"fb879140-59b4-4467-adfe-460df17e72be","Type":"ContainerStarted","Data":"3661da71e23fd3b17fdf859f9f5f9550d0337f8c4b627ed9f9afb43201c9f030"} Jul 11 23:32:10.100708 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.100672 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:32:10.101993 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.101961 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2l9dx" event={"ID":"d7c0b6ae-142a-40a8-ab9b-c56878988bed","Type":"ContainerStarted","Data":"0c9abfa0fda017bdf74cc454fc3072022271442c6ebc42139db4eec494441870"} Jul 11 23:32:10.116883 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.116842 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-v692p" podStartSLOduration=25.271490517 podStartE2EDuration="33.11682982s" totalImagesPullingTime="7.845339303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:10.115816741 +0000 UTC m=+65.771439998" watchObservedRunningTime="2026-07-11 23:32:10.11682982 +0000 UTC m=+65.772453100" Jul 11 23:32:10.487137 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.487112 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-v692p" Jul 11 23:32:10.521453 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.521422 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:32:10.523999 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.523975 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 11 23:32:10.534363 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.534342 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67e8fbc2-1ffc-42d1-8ea6-307da940eda6-metrics-certs\") pod \"network-metrics-daemon-2jq6z\" (UID: \"67e8fbc2-1ffc-42d1-8ea6-307da940eda6\") " pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:32:10.622332 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.622053 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:32:10.622332 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.622142 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:32:10.624802 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.624781 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 11 23:32:10.624994 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.624972 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"image-registry-75bf7ff465-njvd6\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:32:10.635845 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.635823 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80391286-a001-4714-95f7-47d673ca1a6b-original-pull-secret\") pod \"global-pull-secret-syncer-q5tgg\" (UID: \"80391286-a001-4714-95f7-47d673ca1a6b\") " pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:32:10.654157 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.654131 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q5tgg" Jul 11 23:32:10.662554 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.662536 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxtg4\"" Jul 11 23:32:10.665644 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.665614 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:10.670218 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.670202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2jq6z" Jul 11 23:32:10.722802 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.722771 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:32:10.722991 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.722849 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:32:10.722991 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.722907 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:10.724388 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.724292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:32:10.725416 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.724660 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67d5e183-87ab-406e-a885-243f0e2eff99-service-ca-bundle\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:10.725416 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.724732 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:32:10.725416 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.724773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:32:10.725416 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.724804 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:10.726395 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.726370 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1db19fb-9d59-4530-b624-e45f5590ff47-metrics-tls\") pod \"dns-default-7tlrm\" (UID: \"f1db19fb-9d59-4530-b624-e45f5590ff47\") " pod="openshift-dns/dns-default-7tlrm" Jul 11 23:32:10.726723 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.726702 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhm4x\" (UniqueName: \"kubernetes.io/projected/2162ffaf-d5ff-4e93-bc07-02a8db8d7caf-kube-api-access-nhm4x\") pod \"network-check-target-66kvw\" (UID: \"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf\") " pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:32:10.729095 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.728599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-9qnk4\" (UID: \"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:32:10.729095 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.729058 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab0354db-3a99-4cab-b4fb-56d690e36c98-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-czpjh\" (UID: \"ab0354db-3a99-4cab-b4fb-56d690e36c98\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:32:10.729580 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.729539 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67d5e183-87ab-406e-a885-243f0e2eff99-metrics-certs\") pod \"router-default-c7bdd5f5b-nw2lj\" (UID: \"67d5e183-87ab-406e-a885-243f0e2eff99\") " pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:10.729580 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.729559 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/05d45c15-52aa-4126-8ec5-3ab300906ce5-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rdf9q\" (UID: \"05d45c15-52aa-4126-8ec5-3ab300906ce5\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:32:10.767749 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.767719 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:10.794305 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.794266 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q5tgg"] Jul 11 23:32:10.811345 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.811264 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2jq6z"] Jul 11 23:32:10.819103 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.819036 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:10.822558 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:10.822532 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80391286_a001_4714_95f7_47d673ca1a6b.slice/crio-fdc952cac7e6f2c3ec559e24c714350577c6882e43b88f7b80940bef4ad6c549 WatchSource:0}: Error finding container fdc952cac7e6f2c3ec559e24c714350577c6882e43b88f7b80940bef4ad6c549: Status 404 returned error can't find the container with id fdc952cac7e6f2c3ec559e24c714350577c6882e43b88f7b80940bef4ad6c549 Jul 11 23:32:10.823219 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:10.823201 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e8fbc2_1ffc_42d1_8ea6_307da940eda6.slice/crio-4112cf1113eb2580010080f43c81b8321f70e0600b72c18c4893a0af8268fdee WatchSource:0}: Error finding container 4112cf1113eb2580010080f43c81b8321f70e0600b72c18c4893a0af8268fdee: Status 404 returned error can't find the container with id 4112cf1113eb2580010080f43c81b8321f70e0600b72c18c4893a0af8268fdee Jul 11 23:32:10.857141 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.857122 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-h9mt9\"" Jul 11 23:32:10.865295 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.865280 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:32:10.910117 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.910046 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-b74md\"" Jul 11 23:32:10.918655 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.918631 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" Jul 11 23:32:10.931256 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.931230 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dkjr6\"" Jul 11 23:32:10.936950 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.936899 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8jgcr\"" Jul 11 23:32:10.939254 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.939204 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:10.944897 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.944752 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7tlrm" Jul 11 23:32:10.950827 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.950764 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nfccd\"" Jul 11 23:32:10.958445 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.958424 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" Jul 11 23:32:10.960318 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.960217 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:10.968532 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.968012 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qb5vv\"" Jul 11 23:32:10.977234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.977203 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:32:10.977762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.977652 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-6chxn\"" Jul 11 23:32:10.985964 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:10.985844 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" Jul 11 23:32:11.016131 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.011731 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75bf7ff465-njvd6"] Jul 11 23:32:11.019591 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:11.019558 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec49533_1d9f_4e5d_9126_2c2d4a256734.slice/crio-bdfe3f376d102467d974435125e207473a3b7fdf87ebe02b190f4e2fa4011e6a WatchSource:0}: Error finding container bdfe3f376d102467d974435125e207473a3b7fdf87ebe02b190f4e2fa4011e6a: Status 404 returned error can't find the container with id bdfe3f376d102467d974435125e207473a3b7fdf87ebe02b190f4e2fa4011e6a Jul 11 23:32:11.131100 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.131068 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-rdf9q"] Jul 11 23:32:11.133756 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.133699 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" event={"ID":"3ec49533-1d9f-4e5d-9126-2c2d4a256734","Type":"ContainerStarted","Data":"bdfe3f376d102467d974435125e207473a3b7fdf87ebe02b190f4e2fa4011e6a"} Jul 11 23:32:11.136368 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.135772 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2jq6z" event={"ID":"67e8fbc2-1ffc-42d1-8ea6-307da940eda6","Type":"ContainerStarted","Data":"4112cf1113eb2580010080f43c81b8321f70e0600b72c18c4893a0af8268fdee"} Jul 11 23:32:11.138134 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.138047 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q5tgg" event={"ID":"80391286-a001-4714-95f7-47d673ca1a6b","Type":"ContainerStarted","Data":"fdc952cac7e6f2c3ec559e24c714350577c6882e43b88f7b80940bef4ad6c549"} Jul 11 23:32:11.141283 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.141209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:11.141869 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.141849 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2l9dx" event={"ID":"d7c0b6ae-142a-40a8-ab9b-c56878988bed","Type":"ContainerStarted","Data":"793757af2daa36dfae6c65cdba6369c1cf6301455d92b9d92af8939fb2d1bbde"} Jul 11 23:32:11.172229 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.171935 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7tlrm"] Jul 11 23:32:11.176285 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:11.176254 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1db19fb_9d59_4530_b624_e45f5590ff47.slice/crio-679162b4d49fce2614089ac245bcc134b4fe9c25efd4c851b4848dc66ac58625 WatchSource:0}: Error finding container 679162b4d49fce2614089ac245bcc134b4fe9c25efd4c851b4848dc66ac58625: Status 404 returned error can't find the container with id 679162b4d49fce2614089ac245bcc134b4fe9c25efd4c851b4848dc66ac58625 Jul 11 23:32:11.181309 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.181205 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-c7bdd5f5b-nw2lj"] Jul 11 23:32:11.187604 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:11.187580 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d5e183_87ab_406e_a885_243f0e2eff99.slice/crio-ca67235e4b37833f1998ba3cf7eed1d676fde448c6cd16b2673dc5acf9fffd14 WatchSource:0}: Error finding container ca67235e4b37833f1998ba3cf7eed1d676fde448c6cd16b2673dc5acf9fffd14: Status 404 returned error can't find the container with id ca67235e4b37833f1998ba3cf7eed1d676fde448c6cd16b2673dc5acf9fffd14 Jul 11 23:32:11.197721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.197692 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:11.221149 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.221105 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4"] Jul 11 23:32:11.223105 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:11.223083 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca6e6f9c_e4b4_4b6f_8fc8_b04def114b80.slice/crio-4b2329f44fc342ab681cf533cd4a328b994e40ff6a7e85a6fbda4bfdf8e3ecf8 WatchSource:0}: Error finding container 4b2329f44fc342ab681cf533cd4a328b994e40ff6a7e85a6fbda4bfdf8e3ecf8: Status 404 returned error can't find the container with id 4b2329f44fc342ab681cf533cd4a328b994e40ff6a7e85a6fbda4bfdf8e3ecf8 Jul 11 23:32:11.401013 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.400938 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh"] Jul 11 23:32:11.404492 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.404429 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-66kvw"] Jul 11 23:32:11.408652 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:11.408626 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2162ffaf_d5ff_4e93_bc07_02a8db8d7caf.slice/crio-6741f9a7faefa2b8879a1c88d6da40632090e78fd853a549c656af7d0b7ec756 WatchSource:0}: Error finding container 6741f9a7faefa2b8879a1c88d6da40632090e78fd853a549c656af7d0b7ec756: Status 404 returned error can't find the container with id 6741f9a7faefa2b8879a1c88d6da40632090e78fd853a549c656af7d0b7ec756 Jul 11 23:32:11.536211 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.535976 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:11.587850 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.587726 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:11.935362 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.935260 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:11.988290 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:11.988243 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:12.150180 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.150143 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" event={"ID":"3ec49533-1d9f-4e5d-9126-2c2d4a256734","Type":"ContainerStarted","Data":"14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7"} Jul 11 23:32:12.150614 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.150576 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:32:12.152530 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.152477 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" event={"ID":"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80","Type":"ContainerStarted","Data":"4b2329f44fc342ab681cf533cd4a328b994e40ff6a7e85a6fbda4bfdf8e3ecf8"} Jul 11 23:32:12.157919 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.157893 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-66kvw" event={"ID":"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf","Type":"ContainerStarted","Data":"e3f5c4ac45abdc13327c58c552176bdbac89bc2087855d7c43c12ce23dc33ea2"} Jul 11 23:32:12.158028 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.157945 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:32:12.158028 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.157970 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-66kvw" event={"ID":"2162ffaf-d5ff-4e93-bc07-02a8db8d7caf","Type":"ContainerStarted","Data":"6741f9a7faefa2b8879a1c88d6da40632090e78fd853a549c656af7d0b7ec756"} Jul 11 23:32:12.159574 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.159547 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" event={"ID":"ab0354db-3a99-4cab-b4fb-56d690e36c98","Type":"ContainerStarted","Data":"f7e25d40c89e73d199ddb1a95c78a234689b24638b3147297393f9945ebdc2fd"} Jul 11 23:32:12.163805 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.163766 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tlrm" event={"ID":"f1db19fb-9d59-4530-b624-e45f5590ff47","Type":"ContainerStarted","Data":"679162b4d49fce2614089ac245bcc134b4fe9c25efd4c851b4848dc66ac58625"} Jul 11 23:32:12.166811 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.166783 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" event={"ID":"67d5e183-87ab-406e-a885-243f0e2eff99","Type":"ContainerStarted","Data":"b713b3c30e2f4623526a141d650f3f6748e3c40a407290eccbccf6970a109d2d"} Jul 11 23:32:12.166912 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.166834 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" event={"ID":"67d5e183-87ab-406e-a885-243f0e2eff99","Type":"ContainerStarted","Data":"ca67235e4b37833f1998ba3cf7eed1d676fde448c6cd16b2673dc5acf9fffd14"} Jul 11 23:32:12.170271 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.170247 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" event={"ID":"05d45c15-52aa-4126-8ec5-3ab300906ce5","Type":"ContainerStarted","Data":"da8697693b095708a8e229fa1bf1a3d026bafb97371203a3d39ee49c743c8383"} Jul 11 23:32:12.173991 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.173948 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" podStartSLOduration=67.173931027 podStartE2EDuration="1m7.173931027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:12.17333 +0000 UTC m=+67.828953279" watchObservedRunningTime="2026-07-11 23:32:12.173931027 +0000 UTC m=+67.829554282" Jul 11 23:32:12.187984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.187908 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-66kvw" podStartSLOduration=67.18789133 podStartE2EDuration="1m7.18789133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:12.18655556 +0000 UTC m=+67.842178838" watchObservedRunningTime="2026-07-11 23:32:12.18789133 +0000 UTC m=+67.843514584" Jul 11 23:32:12.204548 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.204085 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" podStartSLOduration=35.204073025 podStartE2EDuration="35.204073025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:12.203659342 +0000 UTC m=+67.859282618" watchObservedRunningTime="2026-07-11 23:32:12.204073025 +0000 UTC m=+67.859696278" Jul 11 23:32:12.547390 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.547252 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:12.750377 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.750318 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:12.904659 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.904563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:12.940812 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.939775 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:12.942878 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:12.942847 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:13.167656 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:13.167552 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:13.175768 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:13.175708 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:13.177302 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:13.177147 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-c7bdd5f5b-nw2lj" Jul 11 23:32:14.810017 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.809961 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-596nr"] Jul 11 23:32:14.853209 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.853172 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:14.853680 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.853655 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-596nr"] Jul 11 23:32:14.853825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.853769 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-596nr" Jul 11 23:32:14.857004 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.856855 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 11 23:32:14.857004 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.856887 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 11 23:32:14.857524 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.857445 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xb2tj\"" Jul 11 23:32:14.933670 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.933627 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:14.967606 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:14.967577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbz6z\" (UniqueName: \"kubernetes.io/projected/bb92ba2e-4451-4c21-8bb1-951bdd1b9079-kube-api-access-rbz6z\") pod \"downloads-68c9cbb497-596nr\" (UID: \"bb92ba2e-4451-4c21-8bb1-951bdd1b9079\") " pod="openshift-console/downloads-68c9cbb497-596nr" Jul 11 23:32:15.007120 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:15.007083 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:15.068869 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:15.068798 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbz6z\" (UniqueName: \"kubernetes.io/projected/bb92ba2e-4451-4c21-8bb1-951bdd1b9079-kube-api-access-rbz6z\") pod \"downloads-68c9cbb497-596nr\" (UID: \"bb92ba2e-4451-4c21-8bb1-951bdd1b9079\") " pod="openshift-console/downloads-68c9cbb497-596nr" Jul 11 23:32:15.077142 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:15.077117 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbz6z\" (UniqueName: \"kubernetes.io/projected/bb92ba2e-4451-4c21-8bb1-951bdd1b9079-kube-api-access-rbz6z\") pod \"downloads-68c9cbb497-596nr\" (UID: \"bb92ba2e-4451-4c21-8bb1-951bdd1b9079\") " pod="openshift-console/downloads-68c9cbb497-596nr" Jul 11 23:32:15.125152 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:15.125117 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:15.166826 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:15.166804 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-596nr" Jul 11 23:32:15.769129 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:15.769094 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:15.854919 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:15.854885 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:18.500357 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:18.500324 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-596nr"] Jul 11 23:32:18.521174 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:18.521149 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb92ba2e_4451_4c21_8bb1_951bdd1b9079.slice/crio-027f4fa82ed2b747bc9a557f716e40a257259ea16a8cc1de3b595d148d57c626 WatchSource:0}: Error finding container 027f4fa82ed2b747bc9a557f716e40a257259ea16a8cc1de3b595d148d57c626: Status 404 returned error can't find the container with id 027f4fa82ed2b747bc9a557f716e40a257259ea16a8cc1de3b595d148d57c626 Jul 11 23:32:19.135856 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.135820 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:19.201222 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.200717 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" event={"ID":"ab0354db-3a99-4cab-b4fb-56d690e36c98","Type":"ContainerStarted","Data":"dab5ee77f2f70013c097545cb6650b748bece404ff1429dbc92176317987bb29"} Jul 11 23:32:19.201222 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.200776 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" event={"ID":"ab0354db-3a99-4cab-b4fb-56d690e36c98","Type":"ContainerStarted","Data":"2572878e2a8ffc8cc1c095183cd969b3b0238979140dd5e7130661f0b96f3131"} Jul 11 23:32:19.203913 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.203855 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q5tgg" event={"ID":"80391286-a001-4714-95f7-47d673ca1a6b","Type":"ContainerStarted","Data":"4e8ff9ed5f0d3b7d2bf6de31f3c5b230272a2726b874b1d8d7189ea4577a9a99"} Jul 11 23:32:19.206204 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.206183 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tlrm" event={"ID":"f1db19fb-9d59-4530-b624-e45f5590ff47","Type":"ContainerStarted","Data":"cb06536e5e4d08e26341f6b369c51568fe33ee622b92397e80da44e61f63da4e"} Jul 11 23:32:19.206328 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.206235 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7tlrm" Jul 11 23:32:19.206328 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.206262 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tlrm" event={"ID":"f1db19fb-9d59-4530-b624-e45f5590ff47","Type":"ContainerStarted","Data":"130da3b85db73ccf0d3594c6c975a1a2c38a70743491b59a42c19faaffba603d"} Jul 11 23:32:19.208293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.208268 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2jq6z" event={"ID":"67e8fbc2-1ffc-42d1-8ea6-307da940eda6","Type":"ContainerStarted","Data":"d8dbe5d3c43f8d78493c766383070b8f893ea1261a0b5c8081cff5ed93c4045e"} Jul 11 23:32:19.208388 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.208312 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2jq6z" event={"ID":"67e8fbc2-1ffc-42d1-8ea6-307da940eda6","Type":"ContainerStarted","Data":"8c5e325e6c902c14d0ec76f6590da5ae443d172ab815ec4ff4aa5e3d2a58ad4e"} Jul 11 23:32:19.211555 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.211533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2l9dx" event={"ID":"d7c0b6ae-142a-40a8-ab9b-c56878988bed","Type":"ContainerStarted","Data":"93313d68679cf0391814cc9551f9ba0ec2329ef9b83493a644da753f16daca01"} Jul 11 23:32:19.213234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.213210 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" event={"ID":"ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80","Type":"ContainerStarted","Data":"0902898c930549a76078aa99bafde353cf606967e170359ed1ad1f8a07aea69a"} Jul 11 23:32:19.214851 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.214823 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-596nr" event={"ID":"bb92ba2e-4451-4c21-8bb1-951bdd1b9079","Type":"ContainerStarted","Data":"027f4fa82ed2b747bc9a557f716e40a257259ea16a8cc1de3b595d148d57c626"} Jul 11 23:32:19.216465 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.216445 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" event={"ID":"05d45c15-52aa-4126-8ec5-3ab300906ce5","Type":"ContainerStarted","Data":"6d8d8649585c488a6eed45f4f54ac97ce409064e88fe7bbbbbd4b5e376d693f6"} Jul 11 23:32:19.220984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.220592 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-czpjh" podStartSLOduration=34.103341281 podStartE2EDuration="41.220577082s" totalImagesPullingTime="7.117235801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:19.220139953 +0000 UTC m=+74.875763206" watchObservedRunningTime="2026-07-11 23:32:19.220577082 +0000 UTC m=+74.876200321" Jul 11 23:32:19.227253 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.227191 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:19.241465 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.241214 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q5tgg" podStartSLOduration=66.491502883 podStartE2EDuration="1m14.24119649s" totalImagesPullingTime="7.749693607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:19.24008126 +0000 UTC m=+74.895704514" watchObservedRunningTime="2026-07-11 23:32:19.24119649 +0000 UTC m=+74.896819744" Jul 11 23:32:19.258886 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.258160 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-rdf9q" podStartSLOduration=35.209172246 podStartE2EDuration="42.258142144s" totalImagesPullingTime="7.048969898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:19.257409553 +0000 UTC m=+74.913032833" watchObservedRunningTime="2026-07-11 23:32:19.258142144 +0000 UTC m=+74.913765398" Jul 11 23:32:19.279275 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.279238 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2jq6z" podStartSLOduration=68.957678025 podStartE2EDuration="1m15.279225168s" totalImagesPullingTime="6.321547143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:19.278107254 +0000 UTC m=+74.933730530" watchObservedRunningTime="2026-07-11 23:32:19.279225168 +0000 UTC m=+74.934848443" Jul 11 23:32:19.293005 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.292933 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:19.302771 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.302469 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2l9dx" podStartSLOduration=17.201079372 podStartE2EDuration="29.302453685s" totalImagesPullingTime="12.101374313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-11 23:31:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:19.300695724 +0000 UTC m=+74.956318979" watchObservedRunningTime="2026-07-11 23:32:19.302453685 +0000 UTC m=+74.958076941" Jul 11 23:32:19.326313 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.326276 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7tlrm" podStartSLOduration=34.320460913 podStartE2EDuration="41.326262935s" totalImagesPullingTime="7.005802022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:19.324945248 +0000 UTC m=+74.980568527" watchObservedRunningTime="2026-07-11 23:32:19.326262935 +0000 UTC m=+74.981886199" Jul 11 23:32:19.345223 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.345181 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-9qnk4" podStartSLOduration=33.994359542 podStartE2EDuration="41.34516531s" totalImagesPullingTime="7.350805768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:19.343636108 +0000 UTC m=+74.999259389" watchObservedRunningTime="2026-07-11 23:32:19.34516531 +0000 UTC m=+75.000788563" Jul 11 23:32:19.376573 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.376547 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:19.436022 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.435980 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:19.515415 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.515356 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:19.567519 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:19.567475 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:20.944723 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:20.944672 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:20.993401 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:20.993371 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:21.243142 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:21.243060 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:21.294231 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:21.294196 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:21.453847 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:21.453815 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:21.530529 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:21.530432 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:22.216840 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:22.216806 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:22.316826 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:22.316792 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:24.311879 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:24.311829 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:24.413494 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:24.413457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:24.513118 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:24.513084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:24.613131 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:24.613052 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:24.712868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:24.712821 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:25.371423 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:25.371381 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:25.470801 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:25.470771 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:27.674858 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:27.674723 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:28.370957 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:28.370919 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:28.472983 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:28.472947 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:29.224212 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.224181 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7tlrm" Jul 11 23:32:29.392092 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.392051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:29.486308 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.486137 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:29.588332 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.588286 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:29.706466 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.706432 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:29.783288 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.783210 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5wq7w"] Jul 11 23:32:29.787961 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.787937 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.792625 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.792601 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 11 23:32:29.792753 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.792638 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 11 23:32:29.792753 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.792713 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 11 23:32:29.792753 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.792724 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 11 23:32:29.794199 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.794181 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j7gjb\"" Jul 11 23:32:29.817493 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.817461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:29.884987 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.884954 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-textfile\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885139 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.884996 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9bg\" (UniqueName: \"kubernetes.io/projected/b23100f6-9023-49d8-b503-a05c08a0c73b-kube-api-access-dx9bg\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885139 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.885028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-sys\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885139 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.885080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885139 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.885123 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-wtmp\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.885175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.885219 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b23100f6-9023-49d8-b503-a05c08a0c73b-metrics-client-ca\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.885267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-root\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.885355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.885300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.911471 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.911436 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:29.985777 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.985750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.985919 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.985799 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b23100f6-9023-49d8-b503-a05c08a0c73b-metrics-client-ca\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.985919 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:32:29.985904 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 11 23:32:29.986038 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:32:29.985980 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls podName:b23100f6-9023-49d8-b503-a05c08a0c73b nodeName:}" failed. No retries permitted until 2026-07-11 23:32:30.485959821 +0000 UTC m=+86.141583055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls") pod "node-exporter-5wq7w" (UID: "b23100f6-9023-49d8-b503-a05c08a0c73b") : secret "node-exporter-tls" not found Jul 11 23:32:29.986038 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986002 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-root\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986157 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986034 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986157 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-textfile\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986157 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986089 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-root\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986157 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986109 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9bg\" (UniqueName: \"kubernetes.io/projected/b23100f6-9023-49d8-b503-a05c08a0c73b-kube-api-access-dx9bg\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986157 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-sys\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986369 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986185 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986369 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986236 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-wtmp\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986369 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986357 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-sys\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986538 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-wtmp\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986538 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986459 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b23100f6-9023-49d8-b503-a05c08a0c73b-metrics-client-ca\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986665 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986645 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-textfile\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.986717 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.986696 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.988895 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.988872 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:29.994433 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:29.994412 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9bg\" (UniqueName: \"kubernetes.io/projected/b23100f6-9023-49d8-b503-a05c08a0c73b-kube-api-access-dx9bg\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:30.087984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.087919 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:30.200599 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.200565 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:30.250775 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.250746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:30.301311 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.301273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:30.402406 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.402378 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:30.489666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.489633 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:30.489816 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:32:30.489791 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 11 23:32:30.489883 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:32:30.489861 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls podName:b23100f6-9023-49d8-b503-a05c08a0c73b nodeName:}" failed. No retries permitted until 2026-07-11 23:32:31.489841501 +0000 UTC m=+87.145464745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls") pod "node-exporter-5wq7w" (UID: "b23100f6-9023-49d8-b503-a05c08a0c73b") : secret "node-exporter-tls" not found Jul 11 23:32:30.600053 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.600009 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:30.722111 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.722026 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:30.801068 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:30.801012 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:31.280489 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:31.280446 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:31.372105 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:31.372051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:31.498683 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:31.498650 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:31.501445 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:31.501383 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b23100f6-9023-49d8-b503-a05c08a0c73b-node-exporter-tls\") pod \"node-exporter-5wq7w\" (UID: \"b23100f6-9023-49d8-b503-a05c08a0c73b\") " pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:31.541097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:31.540997 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:31.598963 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:31.598937 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5wq7w" Jul 11 23:32:32.671807 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:32.671771 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:32.771091 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:32.771033 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:32.870968 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:32.870940 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:33.083273 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:33.083179 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:33.180343 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:33.180313 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:32:33.436650 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:33.436618 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:33.539895 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:33.539858 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:34.495975 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:34.495935 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:34.594855 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:34.594817 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:34.732844 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:34.732819 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb23100f6_9023_49d8_b503_a05c08a0c73b.slice/crio-42d503c5761e5a18d745924e8fbe35c296f030a37fe4dabecd026691e879b37a WatchSource:0}: Error finding container 42d503c5761e5a18d745924e8fbe35c296f030a37fe4dabecd026691e879b37a: Status 404 returned error can't find the container with id 42d503c5761e5a18d745924e8fbe35c296f030a37fe4dabecd026691e879b37a Jul 11 23:32:35.234110 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.234083 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:35.289559 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.289463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-596nr" event={"ID":"bb92ba2e-4451-4c21-8bb1-951bdd1b9079","Type":"ContainerStarted","Data":"f1d2d815740c5e761fc3bda641547f19cc5fb8dfdf7743948956be4f12e56e0b"} Jul 11 23:32:35.289772 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.289753 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-596nr" Jul 11 23:32:35.291576 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.291494 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wq7w" event={"ID":"b23100f6-9023-49d8-b503-a05c08a0c73b","Type":"ContainerStarted","Data":"42d503c5761e5a18d745924e8fbe35c296f030a37fe4dabecd026691e879b37a"} Jul 11 23:32:35.306938 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.306895 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-596nr" podStartSLOduration=4.846786775 podStartE2EDuration="21.306877464s" totalImagesPullingTime="16.460090689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:35.306316216 +0000 UTC m=+90.961939469" watchObservedRunningTime="2026-07-11 23:32:35.306877464 +0000 UTC m=+90.962500719" Jul 11 23:32:35.307702 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.307678 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-596nr" Jul 11 23:32:35.818789 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.818753 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:35.920008 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.919975 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:35.974723 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.974693 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:32:35.979896 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.979874 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:35.982984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.982643 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 11 23:32:35.983200 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.983001 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 11 23:32:35.983624 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.983606 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 11 23:32:35.983899 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.983629 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 11 23:32:35.984030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.983650 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g8ndl\"" Jul 11 23:32:35.984105 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.984090 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 11 23:32:35.984165 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.984103 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 11 23:32:35.984220 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.983693 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 11 23:32:35.984635 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.984585 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 11 23:32:35.986054 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.985897 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e1uq84n6uhcg8\"" Jul 11 23:32:35.986054 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.985934 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 11 23:32:35.986482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.986462 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 11 23:32:35.986591 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.986476 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 11 23:32:35.986591 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.986562 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 11 23:32:35.986698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.986670 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 11 23:32:35.987921 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.987897 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 11 23:32:35.991662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.991641 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 11 23:32:35.994657 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.994635 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:32:35.996057 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:35.995919 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:36.036323 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036296 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036469 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036334 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036469 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036361 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036469 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036570 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036601 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036776 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036636 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036776 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036676 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036776 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036755 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036917 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036806 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036917 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036832 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.036917 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.037077 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036947 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.037077 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036977 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.037077 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.036995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.037077 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.037021 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.037077 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.037057 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.037319 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.037081 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt94v\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-kube-api-access-xt94v\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.037319 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.037125 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.137680 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.137647 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.137813 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.137690 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.137813 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.137721 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.137813 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.137748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.137813 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.137785 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138020 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.137943 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138020 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138075 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138113 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138263 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138158 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138263 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138263 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138263 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138460 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138281 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138460 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138460 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138460 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138460 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt94v\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-kube-api-access-xt94v\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138737 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138463 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138737 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138679 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138845 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138809 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.138952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.138904 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.140727 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.139886 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.141122 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.141100 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.142633 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.142332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.142633 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.142362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.142633 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.142523 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.143196 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.142973 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.143418 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.143399 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.145645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.145587 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config-out\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.146098 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.146065 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-web-config\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.146183 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.146124 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.146852 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.146821 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.147524 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.147464 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.147639 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.147617 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.148728 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.148706 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.148832 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.148759 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.149169 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.149146 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.156372 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.156346 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt94v\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-kube-api-access-xt94v\") pod \"prometheus-k8s-0\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.295363 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.295331 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:32:36.298494 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.298460 2566 generic.go:356] "Generic (PLEG): container finished" podID="b23100f6-9023-49d8-b503-a05c08a0c73b" containerID="2c913fca7f6bccaac1325357d3c074f15129ebf14a977fd594d1918003d64304" exitCode=0 Jul 11 23:32:36.298657 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.298542 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wq7w" event={"ID":"b23100f6-9023-49d8-b503-a05c08a0c73b","Type":"ContainerDied","Data":"2c913fca7f6bccaac1325357d3c074f15129ebf14a977fd594d1918003d64304"} Jul 11 23:32:36.498095 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:36.498064 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:32:36.501004 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:32:36.500933 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb4fa18b_a59b_4bdd_86df_f303c67cbef8.slice/crio-fa616bdd1ae4041b4d374f3d5ad32c9282b38fb154bd0cd362d20422cc48252c WatchSource:0}: Error finding container fa616bdd1ae4041b4d374f3d5ad32c9282b38fb154bd0cd362d20422cc48252c: Status 404 returned error can't find the container with id fa616bdd1ae4041b4d374f3d5ad32c9282b38fb154bd0cd362d20422cc48252c Jul 11 23:32:37.306648 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:37.306606 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wq7w" event={"ID":"b23100f6-9023-49d8-b503-a05c08a0c73b","Type":"ContainerStarted","Data":"ef3e0ef893993ac90e1fb98062982a27bc0e56606ad07d21f1680f448631140a"} Jul 11 23:32:37.307198 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:37.306674 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wq7w" event={"ID":"b23100f6-9023-49d8-b503-a05c08a0c73b","Type":"ContainerStarted","Data":"04f07e0fb2a67afe59ceec088894b990a8d7fe4f2401d2f586b3102b3ce79908"} Jul 11 23:32:37.308202 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:37.308175 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"fa616bdd1ae4041b4d374f3d5ad32c9282b38fb154bd0cd362d20422cc48252c"} Jul 11 23:32:37.328210 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:37.328152 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5wq7w" podStartSLOduration=7.065466704 podStartE2EDuration="8.328131847s" totalImagesPullingTime="1.185089257s" totalInitContainerRuntime="77.575886ms" isStatefulPod=true podCreationTimestamp="2026-07-11 23:32:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:37.324706938 +0000 UTC m=+92.980330216" watchObservedRunningTime="2026-07-11 23:32:37.328131847 +0000 UTC m=+92.983755102" Jul 11 23:32:37.400666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:37.400618 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75bf7ff465-njvd6"] Jul 11 23:32:37.488945 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:37.488900 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:37.549845 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:37.549806 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:38.315522 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:38.315476 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} Jul 11 23:32:38.340912 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:38.340883 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:38.428600 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:38.428524 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:38.578463 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:38.578426 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:38.626559 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:38.626493 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:39.321864 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:39.321834 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" exitCode=0 Jul 11 23:32:39.322311 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:39.321925 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} Jul 11 23:32:43.168790 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:43.168746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:43.178188 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:43.178164 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-66kvw" Jul 11 23:32:43.269001 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:43.268918 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:44.347530 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:44.347482 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} Jul 11 23:32:44.347961 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:44.347581 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} Jul 11 23:32:46.359406 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:46.359334 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} Jul 11 23:32:46.359776 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:46.359428 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} Jul 11 23:32:46.359776 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:46.359465 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} Jul 11 23:32:46.359776 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:46.359490 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerStarted","Data":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} Jul 11 23:32:46.388112 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:46.388062 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.4977771779999998 podStartE2EDuration="11.38804613s" totalImagesPullingTime="7.878342921s" totalInitContainerRuntime="11.926031ms" isStatefulPod=false podCreationTimestamp="2026-07-11 23:32:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:32:46.385462001 +0000 UTC m=+102.041085286" watchObservedRunningTime="2026-07-11 23:32:46.38804613 +0000 UTC m=+102.043669382" Jul 11 23:32:47.346656 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:47.346616 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:50.647364 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:50.647326 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:50.747780 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:50.747751 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:50.801076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:50.801032 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:50.944270 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:50.944191 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:50.995373 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:50.995343 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:32:51.295984 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:32:51.295910 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:02.426303 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.426263 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" podUID="3ec49533-1d9f-4e5d-9126-2c2d4a256734" containerName="registry" containerID="cri-o://14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7" gracePeriod=30 Jul 11 23:33:02.694144 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.694092 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:33:02.847520 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847472 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-bound-sa-token\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.847648 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847523 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-trusted-ca\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.847648 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847621 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-certificates\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.847753 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847673 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-installation-pull-secrets\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.847753 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847697 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.847753 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847733 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-image-registry-private-configuration\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.847905 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847768 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec49533-1d9f-4e5d-9126-2c2d4a256734-ca-trust-extracted\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.847905 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847806 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7k8tl\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-kube-api-access-7k8tl\") pod \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\" (UID: \"3ec49533-1d9f-4e5d-9126-2c2d4a256734\") " Jul 11 23:33:02.848009 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.847952 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:02.848066 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.848038 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:02.848118 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.848078 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-trusted-ca\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:02.850183 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.850153 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-kube-api-access-7k8tl" (OuterVolumeSpecName: "kube-api-access-7k8tl") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "kube-api-access-7k8tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:33:02.850292 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.850174 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:02.850292 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.850202 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:33:02.850292 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.850271 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:33:02.850292 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.850277 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:02.857113 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.857089 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec49533-1d9f-4e5d-9126-2c2d4a256734-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3ec49533-1d9f-4e5d-9126-2c2d4a256734" (UID: "3ec49533-1d9f-4e5d-9126-2c2d4a256734"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 23:33:02.948795 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.948741 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ec49533-1d9f-4e5d-9126-2c2d4a256734-ca-trust-extracted\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:02.948795 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.948763 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k8tl\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-kube-api-access-7k8tl\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:02.948795 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.948777 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-bound-sa-token\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:02.948795 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.948792 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-certificates\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:02.948972 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.948807 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-installation-pull-secrets\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:02.948972 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.948819 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ec49533-1d9f-4e5d-9126-2c2d4a256734-registry-tls\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:02.948972 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:02.948834 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ec49533-1d9f-4e5d-9126-2c2d4a256734-image-registry-private-configuration\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:03.423234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.423207 2566 generic.go:356] "Generic (PLEG): container finished" podID="3ec49533-1d9f-4e5d-9126-2c2d4a256734" containerID="14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7" exitCode=0 Jul 11 23:33:03.423350 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.423254 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" Jul 11 23:33:03.423350 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.423295 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" event={"ID":"3ec49533-1d9f-4e5d-9126-2c2d4a256734","Type":"ContainerDied","Data":"14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7"} Jul 11 23:33:03.423350 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.423343 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-75bf7ff465-njvd6" event={"ID":"3ec49533-1d9f-4e5d-9126-2c2d4a256734","Type":"ContainerDied","Data":"bdfe3f376d102467d974435125e207473a3b7fdf87ebe02b190f4e2fa4011e6a"} Jul 11 23:33:03.423452 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.423363 2566 scope.go:117] "RemoveContainer" containerID="14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7" Jul 11 23:33:03.432206 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.432021 2566 scope.go:117] "RemoveContainer" containerID="14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7" Jul 11 23:33:03.432417 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:03.432272 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7\": container with ID starting with 14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7 not found: ID does not exist" containerID="14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7" Jul 11 23:33:03.432417 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.432299 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7"} err="failed to get container status \"14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7\": rpc error: code = NotFound desc = could not find container \"14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7\": container with ID starting with 14e0049d1a4f81fc3b852897d4fd326d5d8befb801a2c3c83c8065ff0b2440d7 not found: ID does not exist" Jul 11 23:33:03.443989 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.443961 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75bf7ff465-njvd6"] Jul 11 23:33:03.448899 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:03.448879 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75bf7ff465-njvd6"] Jul 11 23:33:04.434092 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:04.434066 2566 generic.go:356] "Generic (PLEG): container finished" podID="22088323-b6b3-4e92-bc92-7b44ad14c8dc" containerID="98fd543da5e27360f4861992fce6a0bd87868cb2ed2c8274afdeb35c8439cbe3" exitCode=0 Jul 11 23:33:04.434371 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:04.434139 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" event={"ID":"22088323-b6b3-4e92-bc92-7b44ad14c8dc","Type":"ContainerDied","Data":"98fd543da5e27360f4861992fce6a0bd87868cb2ed2c8274afdeb35c8439cbe3"} Jul 11 23:33:04.434455 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:04.434442 2566 scope.go:117] "RemoveContainer" containerID="98fd543da5e27360f4861992fce6a0bd87868cb2ed2c8274afdeb35c8439cbe3" Jul 11 23:33:04.807021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:04.806992 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec49533-1d9f-4e5d-9126-2c2d4a256734" path="/var/lib/kubelet/pods/3ec49533-1d9f-4e5d-9126-2c2d4a256734/volumes" Jul 11 23:33:05.440032 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:05.440004 2566 generic.go:356] "Generic (PLEG): container finished" podID="4f55f05e-4a71-41c8-99fc-a4c481d7d2e7" containerID="bf682f916cb232264cf7d3841aaec94781c942b77fe44fc41f02ade5c4fe6df6" exitCode=0 Jul 11 23:33:05.440431 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:05.440086 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" event={"ID":"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7","Type":"ContainerDied","Data":"bf682f916cb232264cf7d3841aaec94781c942b77fe44fc41f02ade5c4fe6df6"} Jul 11 23:33:05.440531 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:05.440436 2566 scope.go:117] "RemoveContainer" containerID="bf682f916cb232264cf7d3841aaec94781c942b77fe44fc41f02ade5c4fe6df6" Jul 11 23:33:05.441888 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:05.441867 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-zvwpm" event={"ID":"22088323-b6b3-4e92-bc92-7b44ad14c8dc","Type":"ContainerStarted","Data":"1819ef27318344582d47d83dbb932aaa95d7418d51bf89c9c68f84bd9ab16951"} Jul 11 23:33:05.952304 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:05.952273 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:06.002091 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:06.002059 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:06.448023 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:06.447989 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-kqh7m" event={"ID":"4f55f05e-4a71-41c8-99fc-a4c481d7d2e7","Type":"ContainerStarted","Data":"e7c2669d0dd53609274d9c691ab1c2677ed49383b8635e543be90ace3ae3c61a"} Jul 11 23:33:07.749099 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:07.749053 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:10.470228 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:10.470193 2566 generic.go:356] "Generic (PLEG): container finished" podID="6057afd9-b8a3-4f14-9f90-af1beeba6e3a" containerID="55bc1204def07c8182af40707f9e0099e1714c6b6d4af263b90c676fb33681c1" exitCode=0 Jul 11 23:33:10.470693 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:10.470256 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" event={"ID":"6057afd9-b8a3-4f14-9f90-af1beeba6e3a","Type":"ContainerDied","Data":"55bc1204def07c8182af40707f9e0099e1714c6b6d4af263b90c676fb33681c1"} Jul 11 23:33:10.470693 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:10.470578 2566 scope.go:117] "RemoveContainer" containerID="55bc1204def07c8182af40707f9e0099e1714c6b6d4af263b90c676fb33681c1" Jul 11 23:33:11.307870 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:11.307839 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7bdd5f5b-nw2lj_67d5e183-87ab-406e-a885-243f0e2eff99/router/0.log" Jul 11 23:33:11.318413 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:11.318386 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kqrpg_fce62751-789d-4923-8302-a07368e66a11/serve-healthcheck-canary/0.log" Jul 11 23:33:11.476422 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:11.476394 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-4qd7j" event={"ID":"6057afd9-b8a3-4f14-9f90-af1beeba6e3a","Type":"ContainerStarted","Data":"c1c5f7fec8d9047800d2495cd3e2467da69ce5ec4aa225a5f9f77ec23038f617"} Jul 11 23:33:13.199810 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:13.199772 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:13.300343 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:13.300309 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:13.348165 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:13.348134 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:36.295546 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:36.295486 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:36.310224 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:36.310201 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:36.589682 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:36.589615 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:45.809347 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:45.809312 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:45.864036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:45.863999 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:53.413832 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:53.413796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:53.530878 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:53.530849 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:53.579159 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:53.579136 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:54.129481 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.129449 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:54.272522 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.272481 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:33:54.272810 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.272777 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="prometheus" containerID="cri-o://ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" gracePeriod=600 Jul 11 23:33:54.272810 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.272793 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="thanos-sidecar" containerID="cri-o://3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" gracePeriod=600 Jul 11 23:33:54.273030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.272814 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="config-reloader" containerID="cri-o://fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" gracePeriod=600 Jul 11 23:33:54.273030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.272848 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-web" containerID="cri-o://d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" gracePeriod=600 Jul 11 23:33:54.273030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.272795 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy" containerID="cri-o://ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" gracePeriod=600 Jul 11 23:33:54.273030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.272838 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" gracePeriod=600 Jul 11 23:33:54.507457 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.507436 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.615786 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615759 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-kubelet-serving-ca-bundle\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.615916 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615792 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-2\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.615916 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615822 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.615916 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615851 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-thanos-prometheus-http-client-file\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.615916 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615880 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-metrics-client-ca\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.615916 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615908 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-1\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615945 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.615972 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-tls\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616004 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-tls-assets\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config-out\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616065 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-kube-rbac-proxy\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616092 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-web-config\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616125 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-db\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616165 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-0\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616190 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-serving-certs-ca-bundle\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616176 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616215 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt94v\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-kube-api-access-xt94v\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616271 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-metrics-client-certs\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616308 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-grpc-tls\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616339 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-trusted-ca-bundle\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616365 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config\") pod \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\" (UID: \"fb4fa18b-a59b-4bdd-86df-f303c67cbef8\") " Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616408 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:54.616482 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616456 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:54.616995 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616683 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.616995 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616706 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.616995 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.616722 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-metrics-client-ca\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.617152 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.617062 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:54.618698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.618466 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 23:33:54.618698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.618479 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.618698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.618525 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.618698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.618649 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.618971 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.618766 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.618971 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.618780 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-kube-api-access-xt94v" (OuterVolumeSpecName: "kube-api-access-xt94v") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "kube-api-access-xt94v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:33:54.619093 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.619069 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.619151 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.619112 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:54.619203 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.619185 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config" (OuterVolumeSpecName: "config") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.619585 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.619561 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:54.619794 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.619762 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:33:54.620722 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.620696 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.620797 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.620700 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:33:54.620840 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.620814 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.621366 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.621345 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config-out" (OuterVolumeSpecName: "config-out") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 23:33:54.630775 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.630753 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-web-config" (OuterVolumeSpecName: "web-config") pod "fb4fa18b-a59b-4bdd-86df-f303c67cbef8" (UID: "fb4fa18b-a59b-4bdd-86df-f303c67cbef8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:33:54.646882 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646856 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" exitCode=0 Jul 11 23:33:54.646882 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646882 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" exitCode=0 Jul 11 23:33:54.646998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646891 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" exitCode=0 Jul 11 23:33:54.646998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646899 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" exitCode=0 Jul 11 23:33:54.646998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646907 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" exitCode=0 Jul 11 23:33:54.646998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646914 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" exitCode=0 Jul 11 23:33:54.646998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646931 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} Jul 11 23:33:54.646998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.646949 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.647172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.647020 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} Jul 11 23:33:54.647172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.647049 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} Jul 11 23:33:54.647172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.647072 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} Jul 11 23:33:54.647172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.647092 2566 scope.go:117] "RemoveContainer" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.647172 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.647096 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} Jul 11 23:33:54.647334 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.647188 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} Jul 11 23:33:54.647334 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.647219 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fb4fa18b-a59b-4bdd-86df-f303c67cbef8","Type":"ContainerDied","Data":"fa616bdd1ae4041b4d374f3d5ad32c9282b38fb154bd0cd362d20422cc48252c"} Jul 11 23:33:54.655038 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.655024 2566 scope.go:117] "RemoveContainer" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.661692 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.661678 2566 scope.go:117] "RemoveContainer" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.667844 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.667830 2566 scope.go:117] "RemoveContainer" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.669986 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.669968 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:33:54.674631 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.674610 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:33:54.675681 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.675666 2566 scope.go:117] "RemoveContainer" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.681990 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.681975 2566 scope.go:117] "RemoveContainer" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.688738 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.688708 2566 scope.go:117] "RemoveContainer" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.694839 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.694806 2566 scope.go:117] "RemoveContainer" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.695146 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:54.695121 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": container with ID starting with 63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6 not found: ID does not exist" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.695231 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.695161 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} err="failed to get container status \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": rpc error: code = NotFound desc = could not find container \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": container with ID starting with 63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6 not found: ID does not exist" Jul 11 23:33:54.695231 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.695185 2566 scope.go:117] "RemoveContainer" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.695607 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:54.695580 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": container with ID starting with ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33 not found: ID does not exist" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.695683 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.695617 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} err="failed to get container status \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": rpc error: code = NotFound desc = could not find container \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": container with ID starting with ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33 not found: ID does not exist" Jul 11 23:33:54.695683 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.695637 2566 scope.go:117] "RemoveContainer" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.695917 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:54.695898 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": container with ID starting with d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6 not found: ID does not exist" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.695977 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.695921 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} err="failed to get container status \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": rpc error: code = NotFound desc = could not find container \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": container with ID starting with d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6 not found: ID does not exist" Jul 11 23:33:54.695977 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.695936 2566 scope.go:117] "RemoveContainer" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.696165 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:54.696151 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": container with ID starting with 3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec not found: ID does not exist" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.696209 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.696167 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} err="failed to get container status \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": rpc error: code = NotFound desc = could not find container \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": container with ID starting with 3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec not found: ID does not exist" Jul 11 23:33:54.696209 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.696179 2566 scope.go:117] "RemoveContainer" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.696414 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:54.696395 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": container with ID starting with fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae not found: ID does not exist" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.696462 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.696420 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} err="failed to get container status \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": rpc error: code = NotFound desc = could not find container \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": container with ID starting with fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae not found: ID does not exist" Jul 11 23:33:54.696462 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.696436 2566 scope.go:117] "RemoveContainer" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.696692 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:54.696673 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": container with ID starting with ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe not found: ID does not exist" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.696745 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.696698 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} err="failed to get container status \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": rpc error: code = NotFound desc = could not find container \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": container with ID starting with ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe not found: ID does not exist" Jul 11 23:33:54.696745 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.696711 2566 scope.go:117] "RemoveContainer" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.696944 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.696927 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:33:54.697021 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:33:54.696929 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": container with ID starting with b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e not found: ID does not exist" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.697059 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697025 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} err="failed to get container status \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": rpc error: code = NotFound desc = could not find container \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": container with ID starting with b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e not found: ID does not exist" Jul 11 23:33:54.697059 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697037 2566 scope.go:117] "RemoveContainer" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.697231 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697217 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec49533-1d9f-4e5d-9126-2c2d4a256734" containerName="registry" Jul 11 23:33:54.697277 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697236 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec49533-1d9f-4e5d-9126-2c2d4a256734" containerName="registry" Jul 11 23:33:54.697277 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697246 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-thanos" Jul 11 23:33:54.697277 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697243 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} err="failed to get container status \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": rpc error: code = NotFound desc = could not find container \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": container with ID starting with 63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6 not found: ID does not exist" Jul 11 23:33:54.697277 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697264 2566 scope.go:117] "RemoveContainer" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697251 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-thanos" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697318 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="config-reloader" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697327 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="config-reloader" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697333 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="thanos-sidecar" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697342 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="thanos-sidecar" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697352 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-web" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697361 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-web" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697373 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="prometheus" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697381 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="prometheus" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697390 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697397 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697410 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="init-config-reloader" Jul 11 23:33:54.697412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697418 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="init-config-reloader" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697524 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} err="failed to get container status \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": rpc error: code = NotFound desc = could not find container \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": container with ID starting with ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33 not found: ID does not exist" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697540 2566 scope.go:117] "RemoveContainer" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697531 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-web" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697591 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy-thanos" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697598 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="config-reloader" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697608 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="prometheus" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697616 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="kube-rbac-proxy" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697626 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec49533-1d9f-4e5d-9126-2c2d4a256734" containerName="registry" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697635 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" containerName="thanos-sidecar" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697777 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} err="failed to get container status \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": rpc error: code = NotFound desc = could not find container \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": container with ID starting with d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6 not found: ID does not exist" Jul 11 23:33:54.697890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697793 2566 scope.go:117] "RemoveContainer" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.698275 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.697985 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} err="failed to get container status \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": rpc error: code = NotFound desc = could not find container \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": container with ID starting with 3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec not found: ID does not exist" Jul 11 23:33:54.698275 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698007 2566 scope.go:117] "RemoveContainer" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.698275 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698234 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} err="failed to get container status \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": rpc error: code = NotFound desc = could not find container \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": container with ID starting with fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae not found: ID does not exist" Jul 11 23:33:54.698275 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698258 2566 scope.go:117] "RemoveContainer" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.698478 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698455 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} err="failed to get container status \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": rpc error: code = NotFound desc = could not find container \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": container with ID starting with ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe not found: ID does not exist" Jul 11 23:33:54.698549 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698482 2566 scope.go:117] "RemoveContainer" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.698733 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698716 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} err="failed to get container status \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": rpc error: code = NotFound desc = could not find container \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": container with ID starting with b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e not found: ID does not exist" Jul 11 23:33:54.698786 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698733 2566 scope.go:117] "RemoveContainer" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.698946 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698927 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} err="failed to get container status \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": rpc error: code = NotFound desc = could not find container \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": container with ID starting with 63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6 not found: ID does not exist" Jul 11 23:33:54.698990 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.698951 2566 scope.go:117] "RemoveContainer" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.699174 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699156 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} err="failed to get container status \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": rpc error: code = NotFound desc = could not find container \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": container with ID starting with ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33 not found: ID does not exist" Jul 11 23:33:54.699230 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699177 2566 scope.go:117] "RemoveContainer" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.699399 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699383 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} err="failed to get container status \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": rpc error: code = NotFound desc = could not find container \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": container with ID starting with d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6 not found: ID does not exist" Jul 11 23:33:54.699442 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699400 2566 scope.go:117] "RemoveContainer" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.699641 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699622 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} err="failed to get container status \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": rpc error: code = NotFound desc = could not find container \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": container with ID starting with 3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec not found: ID does not exist" Jul 11 23:33:54.699702 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699643 2566 scope.go:117] "RemoveContainer" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.699870 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699852 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} err="failed to get container status \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": rpc error: code = NotFound desc = could not find container \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": container with ID starting with fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae not found: ID does not exist" Jul 11 23:33:54.699914 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.699869 2566 scope.go:117] "RemoveContainer" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.700052 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700037 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} err="failed to get container status \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": rpc error: code = NotFound desc = could not find container \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": container with ID starting with ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe not found: ID does not exist" Jul 11 23:33:54.700052 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700050 2566 scope.go:117] "RemoveContainer" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.700235 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700217 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} err="failed to get container status \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": rpc error: code = NotFound desc = could not find container \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": container with ID starting with b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e not found: ID does not exist" Jul 11 23:33:54.700275 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700235 2566 scope.go:117] "RemoveContainer" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.700422 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700405 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} err="failed to get container status \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": rpc error: code = NotFound desc = could not find container \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": container with ID starting with 63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6 not found: ID does not exist" Jul 11 23:33:54.700422 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700421 2566 scope.go:117] "RemoveContainer" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.700625 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700609 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} err="failed to get container status \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": rpc error: code = NotFound desc = could not find container \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": container with ID starting with ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33 not found: ID does not exist" Jul 11 23:33:54.700671 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700625 2566 scope.go:117] "RemoveContainer" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.700853 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700838 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} err="failed to get container status \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": rpc error: code = NotFound desc = could not find container \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": container with ID starting with d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6 not found: ID does not exist" Jul 11 23:33:54.700901 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.700854 2566 scope.go:117] "RemoveContainer" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.701050 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701033 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} err="failed to get container status \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": rpc error: code = NotFound desc = could not find container \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": container with ID starting with 3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec not found: ID does not exist" Jul 11 23:33:54.701097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701050 2566 scope.go:117] "RemoveContainer" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.701240 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701226 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} err="failed to get container status \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": rpc error: code = NotFound desc = could not find container \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": container with ID starting with fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae not found: ID does not exist" Jul 11 23:33:54.701277 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701240 2566 scope.go:117] "RemoveContainer" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.701416 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701398 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} err="failed to get container status \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": rpc error: code = NotFound desc = could not find container \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": container with ID starting with ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe not found: ID does not exist" Jul 11 23:33:54.701455 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701416 2566 scope.go:117] "RemoveContainer" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.701645 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701626 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} err="failed to get container status \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": rpc error: code = NotFound desc = could not find container \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": container with ID starting with b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e not found: ID does not exist" Jul 11 23:33:54.701703 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701645 2566 scope.go:117] "RemoveContainer" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.701865 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701848 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} err="failed to get container status \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": rpc error: code = NotFound desc = could not find container \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": container with ID starting with 63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6 not found: ID does not exist" Jul 11 23:33:54.701902 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701866 2566 scope.go:117] "RemoveContainer" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.701988 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.701971 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.702360 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.702133 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} err="failed to get container status \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": rpc error: code = NotFound desc = could not find container \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": container with ID starting with ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33 not found: ID does not exist" Jul 11 23:33:54.702360 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.702150 2566 scope.go:117] "RemoveContainer" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.702560 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.702541 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} err="failed to get container status \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": rpc error: code = NotFound desc = could not find container \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": container with ID starting with d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6 not found: ID does not exist" Jul 11 23:33:54.702619 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.702561 2566 scope.go:117] "RemoveContainer" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.702790 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.702774 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} err="failed to get container status \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": rpc error: code = NotFound desc = could not find container \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": container with ID starting with 3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec not found: ID does not exist" Jul 11 23:33:54.702790 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.702791 2566 scope.go:117] "RemoveContainer" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.703023 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703001 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} err="failed to get container status \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": rpc error: code = NotFound desc = could not find container \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": container with ID starting with fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae not found: ID does not exist" Jul 11 23:33:54.703080 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703024 2566 scope.go:117] "RemoveContainer" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.703241 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703222 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} err="failed to get container status \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": rpc error: code = NotFound desc = could not find container \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": container with ID starting with ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe not found: ID does not exist" Jul 11 23:33:54.703284 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703241 2566 scope.go:117] "RemoveContainer" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.703471 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703454 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} err="failed to get container status \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": rpc error: code = NotFound desc = could not find container \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": container with ID starting with b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e not found: ID does not exist" Jul 11 23:33:54.703545 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703471 2566 scope.go:117] "RemoveContainer" containerID="63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6" Jul 11 23:33:54.703702 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703687 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6"} err="failed to get container status \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": rpc error: code = NotFound desc = could not find container \"63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6\": container with ID starting with 63482783e5a0bdb607365c725ddfcaa25e4f0d90c8e0368b93ced6715ddb4da6 not found: ID does not exist" Jul 11 23:33:54.703751 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703702 2566 scope.go:117] "RemoveContainer" containerID="ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33" Jul 11 23:33:54.703948 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703930 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33"} err="failed to get container status \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": rpc error: code = NotFound desc = could not find container \"ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33\": container with ID starting with ef15615aedaf11aed6a2384362544ec20d3fcab19584b09acea5fbd61c71cd33 not found: ID does not exist" Jul 11 23:33:54.703990 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.703949 2566 scope.go:117] "RemoveContainer" containerID="d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6" Jul 11 23:33:54.704143 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704126 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6"} err="failed to get container status \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": rpc error: code = NotFound desc = could not find container \"d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6\": container with ID starting with d400035e901581fda01f6013f9596747d02cec2519193c8da85d90b569f74cd6 not found: ID does not exist" Jul 11 23:33:54.704185 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704144 2566 scope.go:117] "RemoveContainer" containerID="3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec" Jul 11 23:33:54.704349 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704333 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec"} err="failed to get container status \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": rpc error: code = NotFound desc = could not find container \"3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec\": container with ID starting with 3d36ed23da5bd4fd42b6b8489a7da419204709089135aed65bebcadf0945ecec not found: ID does not exist" Jul 11 23:33:54.704349 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704348 2566 scope.go:117] "RemoveContainer" containerID="fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae" Jul 11 23:33:54.704595 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704578 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae"} err="failed to get container status \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": rpc error: code = NotFound desc = could not find container \"fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae\": container with ID starting with fc88749712bb5012bae6d617ace719ba50b2afa10a9d8efc57da4a802e998eae not found: ID does not exist" Jul 11 23:33:54.704666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704595 2566 scope.go:117] "RemoveContainer" containerID="ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe" Jul 11 23:33:54.704815 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704794 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe"} err="failed to get container status \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": rpc error: code = NotFound desc = could not find container \"ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe\": container with ID starting with ae1c0d5e3c8ccb1049fcad116ae3c69110d6bc720435d8b1e8f3d4a0d9dd38fe not found: ID does not exist" Jul 11 23:33:54.704867 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.704821 2566 scope.go:117] "RemoveContainer" containerID="b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e" Jul 11 23:33:54.705074 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705052 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e"} err="failed to get container status \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": rpc error: code = NotFound desc = could not find container \"b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e\": container with ID starting with b297d8dc8cb4f3d482afce9f846764b44a2f4e39d1b3a7915f9c645719da8a2e not found: ID does not exist" Jul 11 23:33:54.705268 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705253 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-g8ndl\"" Jul 11 23:33:54.705342 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705254 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 11 23:33:54.705342 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705254 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 11 23:33:54.705428 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705254 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 11 23:33:54.705428 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705259 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 11 23:33:54.705674 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705659 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 11 23:33:54.705756 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705721 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 11 23:33:54.705811 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705761 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 11 23:33:54.705811 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705783 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 11 23:33:54.705905 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705844 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 11 23:33:54.705905 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705856 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 11 23:33:54.705905 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.705865 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 11 23:33:54.706087 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.706072 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 11 23:33:54.706155 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.706142 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 11 23:33:54.706770 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.706719 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e1uq84n6uhcg8\"" Jul 11 23:33:54.708841 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.708821 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 11 23:33:54.710811 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.710794 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 11 23:33:54.714556 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.714537 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:33:54.717368 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717346 2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-metrics-client-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717372 2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-grpc-tls\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717385 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717396 2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717409 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717422 2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717436 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717449 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717462 2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717473 2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-tls-assets\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717484 2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-config-out\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717495 2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-secret-kube-rbac-proxy\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717545 2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-web-config\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717557 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-db\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717571 2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717582 2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.717602 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.717594 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt94v\" (UniqueName: \"kubernetes.io/projected/fb4fa18b-a59b-4bdd-86df-f303c67cbef8-kube-api-access-xt94v\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:33:54.805412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.805393 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb4fa18b-a59b-4bdd-86df-f303c67cbef8" path="/var/lib/kubelet/pods/fb4fa18b-a59b-4bdd-86df-f303c67cbef8/volumes" Jul 11 23:33:54.818288 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818271 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818343 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-web-config\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818343 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818317 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-config\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818343 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818488 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818348 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818488 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818488 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818428 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11b50933-4fa0-433d-8b27-f2f83f2abe59-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818488 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818527 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qjx\" (UniqueName: \"kubernetes.io/projected/11b50933-4fa0-433d-8b27-f2f83f2abe59-kube-api-access-l7qjx\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818666 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818689 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818739 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818768 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11b50933-4fa0-433d-8b27-f2f83f2abe59-config-out\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818793 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.818825 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.818822 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919241 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919241 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919409 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11b50933-4fa0-433d-8b27-f2f83f2abe59-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919409 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919289 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919409 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919340 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919409 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qjx\" (UniqueName: \"kubernetes.io/projected/11b50933-4fa0-433d-8b27-f2f83f2abe59-kube-api-access-l7qjx\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919620 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919493 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919620 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919563 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919717 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919632 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919717 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919692 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919821 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919729 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919821 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919789 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.919821 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919816 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919870 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11b50933-4fa0-433d-8b27-f2f83f2abe59-config-out\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.919967 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920021 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920236 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920074 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-web-config\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920236 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920099 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-config\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920236 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920126 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920375 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920297 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920375 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920349 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.920476 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920420 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.921001 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.920678 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.922562 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.922252 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11b50933-4fa0-433d-8b27-f2f83f2abe59-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.922562 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.922464 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.922771 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.922566 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.923182 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.923159 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.923314 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.923290 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.923444 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.923421 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-web-config\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.923751 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.923727 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.923845 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.923741 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.924188 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.924161 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.924368 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.924344 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.924420 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.924352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11b50933-4fa0-433d-8b27-f2f83f2abe59-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.924739 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.924723 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-config\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.924941 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.924922 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.924999 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.924984 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11b50933-4fa0-433d-8b27-f2f83f2abe59-config-out\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.925175 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.925157 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11b50933-4fa0-433d-8b27-f2f83f2abe59-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:54.926750 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:54.926734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qjx\" (UniqueName: \"kubernetes.io/projected/11b50933-4fa0-433d-8b27-f2f83f2abe59-kube-api-access-l7qjx\") pod \"prometheus-k8s-0\" (UID: \"11b50933-4fa0-433d-8b27-f2f83f2abe59\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:55.011237 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:55.011211 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:33:55.144654 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:55.144633 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 11 23:33:55.146955 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:33:55.146926 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b50933_4fa0_433d_8b27_f2f83f2abe59.slice/crio-692d5512eda39bb621534fea6009653f6621b8b61ea5562ed446251fea3b4e0e WatchSource:0}: Error finding container 692d5512eda39bb621534fea6009653f6621b8b61ea5562ed446251fea3b4e0e: Status 404 returned error can't find the container with id 692d5512eda39bb621534fea6009653f6621b8b61ea5562ed446251fea3b4e0e Jul 11 23:33:55.656390 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:55.656355 2566 generic.go:356] "Generic (PLEG): container finished" podID="11b50933-4fa0-433d-8b27-f2f83f2abe59" containerID="2339f88b5461712b1282172a9360f8041ad0aa08bc8d5d332b3440e6f8e9b298" exitCode=0 Jul 11 23:33:55.656765 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:55.656442 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerDied","Data":"2339f88b5461712b1282172a9360f8041ad0aa08bc8d5d332b3440e6f8e9b298"} Jul 11 23:33:55.656765 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:55.656575 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerStarted","Data":"692d5512eda39bb621534fea6009653f6621b8b61ea5562ed446251fea3b4e0e"} Jul 11 23:33:56.663815 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:56.663780 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerStarted","Data":"f402ce713b053199853b95fe475817c6caef664d17f32ac0bdd477243f1c3c21"} Jul 11 23:33:56.664147 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:56.663847 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerStarted","Data":"673adbb6b571f067f1c1ebf15447242b14d4cef66031ebe64fffd6262266d018"} Jul 11 23:33:56.664147 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:56.663874 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerStarted","Data":"4c177facbadb4b3da54d8261cbf096e20af052a3a683a2bd4b484eda046e1ff2"} Jul 11 23:33:56.664147 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:56.663905 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerStarted","Data":"f7efd4481a6fa1591051e923bb1a3d01c971b57cd8b2a63662c7ea5593ee208b"} Jul 11 23:33:56.664147 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:56.663929 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerStarted","Data":"e5d381b05b1382f614d11795a4282da75c6bdc42954f311e56e1d6b1f4b2063a"} Jul 11 23:33:56.664147 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:56.663961 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11b50933-4fa0-433d-8b27-f2f83f2abe59","Type":"ContainerStarted","Data":"744355130a745d0ea03f3aad6cc2ce4eb129cd1d8adacbac6d478db779ff6cc9"} Jul 11 23:33:56.692798 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:56.692754 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.672613462 podStartE2EDuration="2.692737633s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.124171ms" isStatefulPod=false podCreationTimestamp="2026-07-11 23:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:33:56.689608814 +0000 UTC m=+172.345232087" watchObservedRunningTime="2026-07-11 23:33:56.692737633 +0000 UTC m=+172.348360888" Jul 11 23:33:57.783858 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:57.783820 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:33:57.832730 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:33:57.832700 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:00.011345 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:00.011315 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:34:03.932873 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:03.932840 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:04.033918 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:04.033876 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:04.134243 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:04.134211 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:05.954848 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:05.954807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:06.052561 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:06.052528 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:06.121688 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:06.121650 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:35.041356 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:35.041320 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:35.136901 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:35.136870 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:41.468220 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:41.468184 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:41.516948 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:41.516918 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:55.012387 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:55.012351 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:34:55.027305 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:55.027283 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:34:55.905075 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:55.905042 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 11 23:34:56.543006 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:56.542968 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:34:57.326243 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:34:57.326213 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:00.135824 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:00.135771 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:00.186524 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:00.186488 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:01.434621 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:01.434583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:01.547646 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:01.547610 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:01.837036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:01.836943 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:01.938858 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:01.938814 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:04.232008 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:04.231973 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:04.642589 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:04.642539 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:06.934287 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:06.934258 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:07.035293 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:07.035262 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:11.838102 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:11.838069 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:14.416647 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:14.416612 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:14.512830 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:14.512794 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:15.535415 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:15.535378 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:15.635568 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:15.635537 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:18.828808 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:18.828772 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:18.930652 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:18.930624 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:21.554006 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:21.553971 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:24.903358 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:24.903318 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:24.955700 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:24.955669 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:27.148980 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:27.148896 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:37.569335 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:37.569293 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:37.671080 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:37.671050 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:38.182543 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.182494 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:38.429782 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.429746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:38.524181 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.524099 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:38.565748 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.565726 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8"] Jul 11 23:35:38.569005 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.568991 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.571465 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.571445 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 11 23:35:38.571870 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.571850 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dzfxt\"" Jul 11 23:35:38.571977 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.571871 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 11 23:35:38.571977 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.571880 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 11 23:35:38.572137 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.572123 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 11 23:35:38.580406 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.580386 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8"] Jul 11 23:35:38.647698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.647673 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znll\" (UniqueName: \"kubernetes.io/projected/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-kube-api-access-2znll\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.647787 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.647704 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.647787 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.647733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-manifests\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.647787 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.647749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.647892 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.647813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-tmp\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.672033 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.672008 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:38.748704 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.748668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znll\" (UniqueName: \"kubernetes.io/projected/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-kube-api-access-2znll\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.748812 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.748715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.748812 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.748746 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-manifests\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.748812 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.748769 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.748812 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.748794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-tmp\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.749132 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.749111 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-manifests\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.751069 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.751050 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-tmp\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.751202 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.751186 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.751268 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.751251 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.757621 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.757599 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znll\" (UniqueName: \"kubernetes.io/projected/63a38ae4-0a39-455b-88a9-02d6c71cf4fe-kube-api-access-2znll\") pod \"opendatahub-operator-controller-manager-6d55578dd8-4wqm8\" (UID: \"63a38ae4-0a39-455b-88a9-02d6c71cf4fe\") " pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.775725 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.775669 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:38.872700 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.872676 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:38.879314 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.879294 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:38.973696 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:38.973644 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:39.012914 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:39.012881 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8"] Jul 11 23:35:39.016026 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:35:39.016003 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a38ae4_0a39_455b_88a9_02d6c71cf4fe.slice/crio-c1a1b5cb139815f81b28acc62b553cd1d967346ec1ff5cd49028aa235be27839 WatchSource:0}: Error finding container c1a1b5cb139815f81b28acc62b553cd1d967346ec1ff5cd49028aa235be27839: Status 404 returned error can't find the container with id c1a1b5cb139815f81b28acc62b553cd1d967346ec1ff5cd49028aa235be27839 Jul 11 23:35:39.038064 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:39.037995 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:39.048773 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:39.048751 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" event={"ID":"63a38ae4-0a39-455b-88a9-02d6c71cf4fe","Type":"ContainerStarted","Data":"c1a1b5cb139815f81b28acc62b553cd1d967346ec1ff5cd49028aa235be27839"} Jul 11 23:35:39.095730 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:39.095706 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:39.204797 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:39.204746 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:39.297695 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:39.297638 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:42.064385 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:42.064356 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" event={"ID":"63a38ae4-0a39-455b-88a9-02d6c71cf4fe","Type":"ContainerStarted","Data":"dbab12f8dfcec695319229a6a1d0d65cfabc07e2a4f4e98fc69d3bf4cd9aecf6"} Jul 11 23:35:43.070580 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:43.070552 2566 generic.go:356] "Generic (PLEG): container finished" podID="63a38ae4-0a39-455b-88a9-02d6c71cf4fe" containerID="dbab12f8dfcec695319229a6a1d0d65cfabc07e2a4f4e98fc69d3bf4cd9aecf6" exitCode=0 Jul 11 23:35:43.070967 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:43.070642 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" event={"ID":"63a38ae4-0a39-455b-88a9-02d6c71cf4fe","Type":"ContainerDied","Data":"dbab12f8dfcec695319229a6a1d0d65cfabc07e2a4f4e98fc69d3bf4cd9aecf6"} Jul 11 23:35:43.626707 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:43.626676 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:43.878967 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:43.878888 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:43.978515 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:43.978457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:44.077262 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:44.077229 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" event={"ID":"63a38ae4-0a39-455b-88a9-02d6c71cf4fe","Type":"ContainerStarted","Data":"bfd53a1b351080d921f359c95867abe07b132f73c002e8ab90533b88f677ba75"} Jul 11 23:35:44.077702 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:44.077355 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:44.101104 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:44.101046 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" podStartSLOduration=1.834474347 podStartE2EDuration="6.101031335s" totalImagesPullingTime="3.279213848s" totalInitContainerRuntime="987.34314ms" isStatefulPod=false podCreationTimestamp="2026-07-11 23:35:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:35:44.099149768 +0000 UTC m=+279.754773049" watchObservedRunningTime="2026-07-11 23:35:44.101031335 +0000 UTC m=+279.756654587" Jul 11 23:35:50.381822 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:50.381768 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:50.483895 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:50.483866 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:55.084522 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:55.084473 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d55578dd8-4wqm8" Jul 11 23:35:57.600666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:57.600629 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:57.728014 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:57.727978 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:58.294097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.294065 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t9jqq"] Jul 11 23:35:58.297595 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.297573 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:58.300318 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.300293 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 11 23:35:58.300318 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.300308 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-sdr5g\"" Jul 11 23:35:58.316353 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.316331 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t9jqq"] Jul 11 23:35:58.337441 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.337405 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:58.397393 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.397354 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e0719d0-9b02-4ded-9537-800d494953a4-cert\") pod \"odh-model-controller-7998c67dc4-t9jqq\" (UID: \"8e0719d0-9b02-4ded-9537-800d494953a4\") " pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:58.397393 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.397395 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqfjc\" (UniqueName: \"kubernetes.io/projected/8e0719d0-9b02-4ded-9537-800d494953a4-kube-api-access-nqfjc\") pod \"odh-model-controller-7998c67dc4-t9jqq\" (UID: \"8e0719d0-9b02-4ded-9537-800d494953a4\") " pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:58.438187 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.438159 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:58.498141 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.498108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e0719d0-9b02-4ded-9537-800d494953a4-cert\") pod \"odh-model-controller-7998c67dc4-t9jqq\" (UID: \"8e0719d0-9b02-4ded-9537-800d494953a4\") " pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:58.498326 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.498143 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfjc\" (UniqueName: \"kubernetes.io/projected/8e0719d0-9b02-4ded-9537-800d494953a4-kube-api-access-nqfjc\") pod \"odh-model-controller-7998c67dc4-t9jqq\" (UID: \"8e0719d0-9b02-4ded-9537-800d494953a4\") " pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:58.498326 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:35:58.498259 2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 11 23:35:58.498434 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:35:58.498334 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e0719d0-9b02-4ded-9537-800d494953a4-cert podName:8e0719d0-9b02-4ded-9537-800d494953a4 nodeName:}" failed. No retries permitted until 2026-07-11 23:35:58.99831635 +0000 UTC m=+294.653939582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8e0719d0-9b02-4ded-9537-800d494953a4-cert") pod "odh-model-controller-7998c67dc4-t9jqq" (UID: "8e0719d0-9b02-4ded-9537-800d494953a4") : secret "odh-model-controller-webhook-cert" not found Jul 11 23:35:58.506874 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.506846 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfjc\" (UniqueName: \"kubernetes.io/projected/8e0719d0-9b02-4ded-9537-800d494953a4-kube-api-access-nqfjc\") pod \"odh-model-controller-7998c67dc4-t9jqq\" (UID: \"8e0719d0-9b02-4ded-9537-800d494953a4\") " pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:58.548080 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.548010 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:58.868826 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.868735 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:58.895939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.895906 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7fv2f"] Jul 11 23:35:58.899268 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.899249 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:58.901690 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.901662 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 11 23:35:58.909636 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.909615 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7fv2f"] Jul 11 23:35:58.969420 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:58.969382 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:59.001977 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.001955 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1-tls-certs\") pod \"model-serving-api-54d89bd5fc-7fv2f\" (UID: \"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1\") " pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:59.002092 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.002005 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7lnl\" (UniqueName: \"kubernetes.io/projected/9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1-kube-api-access-z7lnl\") pod \"model-serving-api-54d89bd5fc-7fv2f\" (UID: \"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1\") " pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:59.002092 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.002079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e0719d0-9b02-4ded-9537-800d494953a4-cert\") pod \"odh-model-controller-7998c67dc4-t9jqq\" (UID: \"8e0719d0-9b02-4ded-9537-800d494953a4\") " pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:59.004434 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.004410 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e0719d0-9b02-4ded-9537-800d494953a4-cert\") pod \"odh-model-controller-7998c67dc4-t9jqq\" (UID: \"8e0719d0-9b02-4ded-9537-800d494953a4\") " pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:59.071143 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.071112 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:35:59.103087 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.103054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1-tls-certs\") pod \"model-serving-api-54d89bd5fc-7fv2f\" (UID: \"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1\") " pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:59.103163 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.103102 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7lnl\" (UniqueName: \"kubernetes.io/projected/9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1-kube-api-access-z7lnl\") pod \"model-serving-api-54d89bd5fc-7fv2f\" (UID: \"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1\") " pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:59.105476 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.105447 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1-tls-certs\") pod \"model-serving-api-54d89bd5fc-7fv2f\" (UID: \"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1\") " pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:59.110267 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.110245 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7lnl\" (UniqueName: \"kubernetes.io/projected/9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1-kube-api-access-z7lnl\") pod \"model-serving-api-54d89bd5fc-7fv2f\" (UID: \"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1\") " pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:59.210441 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.210403 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:35:59.210624 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.210455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:35:59.354650 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.354619 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7fv2f"] Jul 11 23:35:59.360434 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:35:59.360405 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dbde056_e4d7_41aa_9c0b_9f5c206e0ff1.slice/crio-aeebb2297b622ed832e09f2395487915e90126acf0a5facfa1fc6f2a2f978b90 WatchSource:0}: Error finding container aeebb2297b622ed832e09f2395487915e90126acf0a5facfa1fc6f2a2f978b90: Status 404 returned error can't find the container with id aeebb2297b622ed832e09f2395487915e90126acf0a5facfa1fc6f2a2f978b90 Jul 11 23:35:59.374072 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:35:59.374047 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-t9jqq"] Jul 11 23:35:59.376711 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:35:59.376689 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0719d0_9b02_4ded_9537_800d494953a4.slice/crio-2bbd57f7f190828020fd7f2eed0642757859f575a9082d03d331f0223c1859b9 WatchSource:0}: Error finding container 2bbd57f7f190828020fd7f2eed0642757859f575a9082d03d331f0223c1859b9: Status 404 returned error can't find the container with id 2bbd57f7f190828020fd7f2eed0642757859f575a9082d03d331f0223c1859b9 Jul 11 23:36:00.150107 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:00.150067 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" event={"ID":"8e0719d0-9b02-4ded-9537-800d494953a4","Type":"ContainerStarted","Data":"2bbd57f7f190828020fd7f2eed0642757859f575a9082d03d331f0223c1859b9"} Jul 11 23:36:00.151774 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:00.151742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" event={"ID":"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1","Type":"ContainerStarted","Data":"aeebb2297b622ed832e09f2395487915e90126acf0a5facfa1fc6f2a2f978b90"} Jul 11 23:36:03.167936 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.167884 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" event={"ID":"8e0719d0-9b02-4ded-9537-800d494953a4","Type":"ContainerStarted","Data":"b81aae7399c0627fdf4e132f8f3ecdcad01093cc47da7dd45f9140230d2dc317"} Jul 11 23:36:03.168431 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.168023 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:36:03.170009 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.169980 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" event={"ID":"9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1","Type":"ContainerStarted","Data":"69ab1e6fc860e72484451417b51e65eccb6f9471bec07c677502f6c33a8b2283"} Jul 11 23:36:03.170141 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.170074 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:36:03.184173 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.184123 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" podStartSLOduration=1.976386809 podStartE2EDuration="5.184107622s" totalImagesPullingTime="3.207720813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:36:03.183119823 +0000 UTC m=+298.838743078" watchObservedRunningTime="2026-07-11 23:36:03.184107622 +0000 UTC m=+298.839730876" Jul 11 23:36:03.199303 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.199243 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" podStartSLOduration=2.021272741 podStartE2EDuration="5.19922437s" totalImagesPullingTime="3.177951629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:36:03.197383651 +0000 UTC m=+298.853006904" watchObservedRunningTime="2026-07-11 23:36:03.19922437 +0000 UTC m=+298.854847625" Jul 11 23:36:03.364927 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.364884 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:03.632478 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:03.632386 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:04.395940 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.395909 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh"] Jul 11 23:36:04.398041 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.398000 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:04.399834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.399816 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.402593 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.402525 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 11 23:36:04.402593 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.402551 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 11 23:36:04.402593 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.402551 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bvm5c\"" Jul 11 23:36:04.402820 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.402564 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 11 23:36:04.402820 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.402636 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 11 23:36:04.409618 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.409596 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh"] Jul 11 23:36:04.448362 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.448334 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-istio-token\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.448362 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.448362 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-cacerts\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.448535 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.448388 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.448535 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.448430 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86768c7d-0184-4b41-8995-47e58f747855-local-certs\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.448535 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.448486 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pq8j\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-kube-api-access-9pq8j\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.448654 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.448574 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.448654 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.448622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.497491 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.497463 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:04.549803 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.549782 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.549919 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.549820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.549919 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.549881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-istio-token\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.549919 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.549905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-cacerts\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.550039 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.549938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.550039 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.549963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86768c7d-0184-4b41-8995-47e58f747855-local-certs\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.550039 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.549992 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pq8j\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-kube-api-access-9pq8j\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.550573 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.550551 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.552146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.552126 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.552239 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.552183 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-cacerts\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.552566 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.552542 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86768c7d-0184-4b41-8995-47e58f747855-local-certs\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.552672 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.552580 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.561257 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.561229 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-istio-token\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.561698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.561676 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pq8j\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-kube-api-access-9pq8j\") pod \"istiod-openshift-gateway-7574459656-rdpsh\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.673433 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.673362 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:36:04.674987 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.674962 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:36:04.678057 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.678031 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:36:04.682349 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.682320 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:36:04.693034 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.693017 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 11 23:36:04.712806 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.712785 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bvm5c\"" Jul 11 23:36:04.719877 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.719859 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:04.862927 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:04.862908 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh"] Jul 11 23:36:04.865004 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:36:04.864981 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86768c7d_0184_4b41_8995_47e58f747855.slice/crio-c4d3765a7eb3baaa4868a46ea6df30583301e35a0196d3138571cb6935f9951a WatchSource:0}: Error finding container c4d3765a7eb3baaa4868a46ea6df30583301e35a0196d3138571cb6935f9951a: Status 404 returned error can't find the container with id c4d3765a7eb3baaa4868a46ea6df30583301e35a0196d3138571cb6935f9951a Jul 11 23:36:05.181234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:05.181200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" event={"ID":"86768c7d-0184-4b41-8995-47e58f747855","Type":"ContainerStarted","Data":"c4d3765a7eb3baaa4868a46ea6df30583301e35a0196d3138571cb6935f9951a"} Jul 11 23:36:05.432020 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:05.431943 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:06.574343 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.574311 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d758c946-qfznb"] Jul 11 23:36:06.578868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.578851 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.583519 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.583486 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 11 23:36:06.585151 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.585130 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 11 23:36:06.585151 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.585142 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tkxpb\"" Jul 11 23:36:06.585151 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.585157 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 11 23:36:06.585367 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.585174 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 11 23:36:06.585367 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.585196 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 11 23:36:06.598463 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.598445 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d758c946-qfznb"] Jul 11 23:36:06.672491 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.672464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjn2s\" (UniqueName: \"kubernetes.io/projected/983e865e-4d67-41ea-9def-fa43dc2dac79-kube-api-access-qjn2s\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.672621 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.672518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/983e865e-4d67-41ea-9def-fa43dc2dac79-metrics-cert\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.672621 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.672609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/983e865e-4d67-41ea-9def-fa43dc2dac79-cert\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.672712 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.672647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/983e865e-4d67-41ea-9def-fa43dc2dac79-manager-config\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.773857 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.773832 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/983e865e-4d67-41ea-9def-fa43dc2dac79-cert\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.773989 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.773881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/983e865e-4d67-41ea-9def-fa43dc2dac79-manager-config\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.773989 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.773927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjn2s\" (UniqueName: \"kubernetes.io/projected/983e865e-4d67-41ea-9def-fa43dc2dac79-kube-api-access-qjn2s\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.773989 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.773957 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/983e865e-4d67-41ea-9def-fa43dc2dac79-metrics-cert\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.774625 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.774602 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/983e865e-4d67-41ea-9def-fa43dc2dac79-manager-config\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.776233 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.776213 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/983e865e-4d67-41ea-9def-fa43dc2dac79-metrics-cert\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.776308 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.776262 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/983e865e-4d67-41ea-9def-fa43dc2dac79-cert\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.782430 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.782403 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjn2s\" (UniqueName: \"kubernetes.io/projected/983e865e-4d67-41ea-9def-fa43dc2dac79-kube-api-access-qjn2s\") pod \"lws-controller-manager-59d758c946-qfznb\" (UID: \"983e865e-4d67-41ea-9def-fa43dc2dac79\") " pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:06.888538 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:06.888452 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:07.011534 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:07.011496 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59d758c946-qfznb"] Jul 11 23:36:07.013470 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:36:07.013444 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983e865e_4d67_41ea_9def_fa43dc2dac79.slice/crio-0c07f21bc5d43aca3f298aa5d975b5d29c8eb5e7e932975b586b0d0b8c0e13c7 WatchSource:0}: Error finding container 0c07f21bc5d43aca3f298aa5d975b5d29c8eb5e7e932975b586b0d0b8c0e13c7: Status 404 returned error can't find the container with id 0c07f21bc5d43aca3f298aa5d975b5d29c8eb5e7e932975b586b0d0b8c0e13c7 Jul 11 23:36:07.015615 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:07.015599 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 11 23:36:07.190442 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:07.190412 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" event={"ID":"983e865e-4d67-41ea-9def-fa43dc2dac79","Type":"ContainerStarted","Data":"0c07f21bc5d43aca3f298aa5d975b5d29c8eb5e7e932975b586b0d0b8c0e13c7"} Jul 11 23:36:08.128907 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:08.128814 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:09.390323 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:09.390286 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:09.502667 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:09.502636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:09.568632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:09.568603 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:10.206244 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:10.206210 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" event={"ID":"983e865e-4d67-41ea-9def-fa43dc2dac79","Type":"ContainerStarted","Data":"56581c96cb1be2fc46c011dcabf9c1f56ec37e56579cc1113fa44c5a50c6331f"} Jul 11 23:36:10.206394 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:10.206334 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:10.226031 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:10.225990 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" podStartSLOduration=1.734762787 podStartE2EDuration="4.225978031s" totalImagesPullingTime="2.491215244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:36:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:36:10.225641358 +0000 UTC m=+305.881264610" watchObservedRunningTime="2026-07-11 23:36:10.225978031 +0000 UTC m=+305.881601283" Jul 11 23:36:10.247448 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:10.247419 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:10.346608 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:10.346583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:11.063480 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:11.063430 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:11.246494 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:11.246454 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh"] Jul 11 23:36:11.293996 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:11.293959 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:11.495362 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:11.495319 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:12.672754 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:12.672716 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 11 23:36:12.673037 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:12.672781 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 11 23:36:13.223424 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.223387 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" event={"ID":"86768c7d-0184-4b41-8995-47e58f747855","Type":"ContainerStarted","Data":"9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64"} Jul 11 23:36:13.223424 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.223404 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" podUID="86768c7d-0184-4b41-8995-47e58f747855" containerName="discovery" containerID="cri-o://9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64" gracePeriod=30 Jul 11 23:36:13.223697 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.223441 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:13.225033 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.225008 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-rdpsh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 11 23:36:13.225146 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.225049 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" podUID="86768c7d-0184-4b41-8995-47e58f747855" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 11 23:36:13.464426 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.464398 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:13.531672 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.531598 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-cacerts\") pod \"86768c7d-0184-4b41-8995-47e58f747855\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " Jul 11 23:36:13.531672 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.531648 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-ca-configmap\") pod \"86768c7d-0184-4b41-8995-47e58f747855\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " Jul 11 23:36:13.531856 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.531687 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86768c7d-0184-4b41-8995-47e58f747855-local-certs\") pod \"86768c7d-0184-4b41-8995-47e58f747855\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " Jul 11 23:36:13.531856 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.531741 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-istio-token\") pod \"86768c7d-0184-4b41-8995-47e58f747855\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " Jul 11 23:36:13.531856 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.531763 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-dns-cert\") pod \"86768c7d-0184-4b41-8995-47e58f747855\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " Jul 11 23:36:13.531856 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.531814 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pq8j\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-kube-api-access-9pq8j\") pod \"86768c7d-0184-4b41-8995-47e58f747855\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " Jul 11 23:36:13.531856 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.531851 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-kubeconfig\") pod \"86768c7d-0184-4b41-8995-47e58f747855\" (UID: \"86768c7d-0184-4b41-8995-47e58f747855\") " Jul 11 23:36:13.532097 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.532037 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "86768c7d-0184-4b41-8995-47e58f747855" (UID: "86768c7d-0184-4b41-8995-47e58f747855"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 11 23:36:13.532271 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.532160 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-ca-configmap\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:36:13.534449 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.534421 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-cacerts" (OuterVolumeSpecName: "cacerts") pod "86768c7d-0184-4b41-8995-47e58f747855" (UID: "86768c7d-0184-4b41-8995-47e58f747855"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:36:13.534659 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.534629 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "86768c7d-0184-4b41-8995-47e58f747855" (UID: "86768c7d-0184-4b41-8995-47e58f747855"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:36:13.534793 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.534769 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86768c7d-0184-4b41-8995-47e58f747855-local-certs" (OuterVolumeSpecName: "local-certs") pod "86768c7d-0184-4b41-8995-47e58f747855" (UID: "86768c7d-0184-4b41-8995-47e58f747855"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 23:36:13.534888 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.534864 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-istio-token" (OuterVolumeSpecName: "istio-token") pod "86768c7d-0184-4b41-8995-47e58f747855" (UID: "86768c7d-0184-4b41-8995-47e58f747855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:36:13.535071 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.535030 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "86768c7d-0184-4b41-8995-47e58f747855" (UID: "86768c7d-0184-4b41-8995-47e58f747855"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:36:13.535176 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.535157 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-kube-api-access-9pq8j" (OuterVolumeSpecName: "kube-api-access-9pq8j") pod "86768c7d-0184-4b41-8995-47e58f747855" (UID: "86768c7d-0184-4b41-8995-47e58f747855"). InnerVolumeSpecName "kube-api-access-9pq8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:36:13.632606 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.632580 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pq8j\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-kube-api-access-9pq8j\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:36:13.632606 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.632602 2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-kubeconfig\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:36:13.632606 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.632612 2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-cacerts\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:36:13.632791 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.632620 2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/86768c7d-0184-4b41-8995-47e58f747855-local-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:36:13.632791 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.632630 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86768c7d-0184-4b41-8995-47e58f747855-istio-token\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:36:13.632791 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:13.632637 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/86768c7d-0184-4b41-8995-47e58f747855-istio-csr-dns-cert\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:36:14.178262 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.178234 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-t9jqq" Jul 11 23:36:14.180160 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.180140 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7fv2f" Jul 11 23:36:14.230612 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.230583 2566 generic.go:356] "Generic (PLEG): container finished" podID="86768c7d-0184-4b41-8995-47e58f747855" containerID="9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64" exitCode=2 Jul 11 23:36:14.230762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.230620 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" Jul 11 23:36:14.230762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.230660 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" event={"ID":"86768c7d-0184-4b41-8995-47e58f747855","Type":"ContainerDied","Data":"9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64"} Jul 11 23:36:14.230762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.230714 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh" event={"ID":"86768c7d-0184-4b41-8995-47e58f747855","Type":"ContainerDied","Data":"c4d3765a7eb3baaa4868a46ea6df30583301e35a0196d3138571cb6935f9951a"} Jul 11 23:36:14.230762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.230749 2566 scope.go:117] "RemoveContainer" containerID="9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64" Jul 11 23:36:14.248421 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.248400 2566 scope.go:117] "RemoveContainer" containerID="9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64" Jul 11 23:36:14.248739 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:36:14.248720 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64\": container with ID starting with 9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64 not found: ID does not exist" containerID="9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64" Jul 11 23:36:14.248811 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.248745 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64"} err="failed to get container status \"9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64\": rpc error: code = NotFound desc = could not find container \"9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64\": container with ID starting with 9352a73a25aa6ea7c0027c50433fcef2145b673413870cdb5c84fbd6257b8f64 not found: ID does not exist" Jul 11 23:36:14.258702 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.258681 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh"] Jul 11 23:36:14.264406 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.264386 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rdpsh"] Jul 11 23:36:14.806945 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:14.806915 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86768c7d-0184-4b41-8995-47e58f747855" path="/var/lib/kubelet/pods/86768c7d-0184-4b41-8995-47e58f747855/volumes" Jul 11 23:36:21.214374 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:21.214342 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59d758c946-qfznb" Jul 11 23:36:59.052638 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.052553 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:59.148215 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.148181 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:59.164350 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.164323 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hjdck"] Jul 11 23:36:59.164632 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.164620 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86768c7d-0184-4b41-8995-47e58f747855" containerName="discovery" Jul 11 23:36:59.164679 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.164634 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86768c7d-0184-4b41-8995-47e58f747855" containerName="discovery" Jul 11 23:36:59.164713 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.164697 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86768c7d-0184-4b41-8995-47e58f747855" containerName="discovery" Jul 11 23:36:59.167512 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.167486 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" Jul 11 23:36:59.170545 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.170523 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 11 23:36:59.170642 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.170570 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 11 23:36:59.170733 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.170696 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kl5rl\"" Jul 11 23:36:59.178164 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.178131 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hjdck"] Jul 11 23:36:59.212330 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.212310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:36:59.263551 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.263527 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd4t8\" (UniqueName: \"kubernetes.io/projected/6fba839d-05a8-40ba-97e5-812d0135ed91-kube-api-access-rd4t8\") pod \"kuadrant-operator-catalog-hjdck\" (UID: \"6fba839d-05a8-40ba-97e5-812d0135ed91\") " pod="kuadrant-system/kuadrant-operator-catalog-hjdck" Jul 11 23:36:59.364359 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.364306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4t8\" (UniqueName: \"kubernetes.io/projected/6fba839d-05a8-40ba-97e5-812d0135ed91-kube-api-access-rd4t8\") pod \"kuadrant-operator-catalog-hjdck\" (UID: \"6fba839d-05a8-40ba-97e5-812d0135ed91\") " pod="kuadrant-system/kuadrant-operator-catalog-hjdck" Jul 11 23:36:59.372539 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.372495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd4t8\" (UniqueName: \"kubernetes.io/projected/6fba839d-05a8-40ba-97e5-812d0135ed91-kube-api-access-rd4t8\") pod \"kuadrant-operator-catalog-hjdck\" (UID: \"6fba839d-05a8-40ba-97e5-812d0135ed91\") " pod="kuadrant-system/kuadrant-operator-catalog-hjdck" Jul 11 23:36:59.478354 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.478327 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" Jul 11 23:36:59.534638 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.534612 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hjdck"] Jul 11 23:36:59.605349 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.605329 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hjdck"] Jul 11 23:36:59.607542 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:36:59.607495 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fba839d_05a8_40ba_97e5_812d0135ed91.slice/crio-4a7e96011ae67f37d1ad8db173d604082ecbbf6a823eac6781706f0be52312bf WatchSource:0}: Error finding container 4a7e96011ae67f37d1ad8db173d604082ecbbf6a823eac6781706f0be52312bf: Status 404 returned error can't find the container with id 4a7e96011ae67f37d1ad8db173d604082ecbbf6a823eac6781706f0be52312bf Jul 11 23:36:59.744768 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.744741 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xh2gs"] Jul 11 23:36:59.749040 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.749020 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:36:59.754756 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.754735 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xh2gs"] Jul 11 23:36:59.767494 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.767473 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gd9\" (UniqueName: \"kubernetes.io/projected/9abac196-4ee5-4fb9-b41e-01d800152418-kube-api-access-r9gd9\") pod \"kuadrant-operator-catalog-xh2gs\" (UID: \"9abac196-4ee5-4fb9-b41e-01d800152418\") " pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:36:59.867896 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.867873 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gd9\" (UniqueName: \"kubernetes.io/projected/9abac196-4ee5-4fb9-b41e-01d800152418-kube-api-access-r9gd9\") pod \"kuadrant-operator-catalog-xh2gs\" (UID: \"9abac196-4ee5-4fb9-b41e-01d800152418\") " pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:36:59.875399 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:36:59.875374 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gd9\" (UniqueName: \"kubernetes.io/projected/9abac196-4ee5-4fb9-b41e-01d800152418-kube-api-access-r9gd9\") pod \"kuadrant-operator-catalog-xh2gs\" (UID: \"9abac196-4ee5-4fb9-b41e-01d800152418\") " pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:37:00.059166 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:00.059107 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:37:00.194396 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:00.194366 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xh2gs"] Jul 11 23:37:00.233991 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:37:00.233967 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abac196_4ee5_4fb9_b41e_01d800152418.slice/crio-f0944973b4db336b172d439c89032d336ab85ec0bbf29eccec0f36e038f4f81c WatchSource:0}: Error finding container f0944973b4db336b172d439c89032d336ab85ec0bbf29eccec0f36e038f4f81c: Status 404 returned error can't find the container with id f0944973b4db336b172d439c89032d336ab85ec0bbf29eccec0f36e038f4f81c Jul 11 23:37:00.430012 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:00.429979 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" event={"ID":"6fba839d-05a8-40ba-97e5-812d0135ed91","Type":"ContainerStarted","Data":"4a7e96011ae67f37d1ad8db173d604082ecbbf6a823eac6781706f0be52312bf"} Jul 11 23:37:00.431246 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:00.431209 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" event={"ID":"9abac196-4ee5-4fb9-b41e-01d800152418","Type":"ContainerStarted","Data":"f0944973b4db336b172d439c89032d336ab85ec0bbf29eccec0f36e038f4f81c"} Jul 11 23:37:02.442532 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.442484 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" event={"ID":"9abac196-4ee5-4fb9-b41e-01d800152418","Type":"ContainerStarted","Data":"4b7ab4b8f286a5cd5e6e784a04fe1e09130a5d9102ee128a7d61f9dff3b65cc1"} Jul 11 23:37:02.444010 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.443987 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" event={"ID":"6fba839d-05a8-40ba-97e5-812d0135ed91","Type":"ContainerStarted","Data":"685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc"} Jul 11 23:37:02.444125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.444063 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" podUID="6fba839d-05a8-40ba-97e5-812d0135ed91" containerName="registry-server" containerID="cri-o://685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc" gracePeriod=2 Jul 11 23:37:02.456937 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.456888 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" podStartSLOduration=2.059935606 podStartE2EDuration="3.456870796s" totalImagesPullingTime="1.39693519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:02.456218282 +0000 UTC m=+358.111841555" watchObservedRunningTime="2026-07-11 23:37:02.456870796 +0000 UTC m=+358.112494049" Jul 11 23:37:02.471577 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.471481 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" podStartSLOduration=1.451471649 podStartE2EDuration="3.471464442s" totalImagesPullingTime="2.019992793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:02.469870753 +0000 UTC m=+358.125494030" watchObservedRunningTime="2026-07-11 23:37:02.471464442 +0000 UTC m=+358.127087696" Jul 11 23:37:02.689396 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.689374 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" Jul 11 23:37:02.809253 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.809195 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd4t8\" (UniqueName: \"kubernetes.io/projected/6fba839d-05a8-40ba-97e5-812d0135ed91-kube-api-access-rd4t8\") pod \"6fba839d-05a8-40ba-97e5-812d0135ed91\" (UID: \"6fba839d-05a8-40ba-97e5-812d0135ed91\") " Jul 11 23:37:02.811224 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.811195 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fba839d-05a8-40ba-97e5-812d0135ed91-kube-api-access-rd4t8" (OuterVolumeSpecName: "kube-api-access-rd4t8") pod "6fba839d-05a8-40ba-97e5-812d0135ed91" (UID: "6fba839d-05a8-40ba-97e5-812d0135ed91"). InnerVolumeSpecName "kube-api-access-rd4t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:37:02.910599 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:02.910570 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd4t8\" (UniqueName: \"kubernetes.io/projected/6fba839d-05a8-40ba-97e5-812d0135ed91-kube-api-access-rd4t8\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:37:03.449414 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.449382 2566 generic.go:356] "Generic (PLEG): container finished" podID="6fba839d-05a8-40ba-97e5-812d0135ed91" containerID="685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc" exitCode=0 Jul 11 23:37:03.449801 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.449446 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" Jul 11 23:37:03.449801 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.449462 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" event={"ID":"6fba839d-05a8-40ba-97e5-812d0135ed91","Type":"ContainerDied","Data":"685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc"} Jul 11 23:37:03.449801 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.449551 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hjdck" event={"ID":"6fba839d-05a8-40ba-97e5-812d0135ed91","Type":"ContainerDied","Data":"4a7e96011ae67f37d1ad8db173d604082ecbbf6a823eac6781706f0be52312bf"} Jul 11 23:37:03.449801 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.449573 2566 scope.go:117] "RemoveContainer" containerID="685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc" Jul 11 23:37:03.458771 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.458754 2566 scope.go:117] "RemoveContainer" containerID="685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc" Jul 11 23:37:03.459013 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:03.458994 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc\": container with ID starting with 685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc not found: ID does not exist" containerID="685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc" Jul 11 23:37:03.459082 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.459020 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc"} err="failed to get container status \"685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc\": rpc error: code = NotFound desc = could not find container \"685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc\": container with ID starting with 685fd6e6e42b16d14edadd9b5a0c487e3e199678fa672ff5fa1eb01d1e8180bc not found: ID does not exist" Jul 11 23:37:03.470247 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.470225 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hjdck"] Jul 11 23:37:03.475108 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:03.475090 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hjdck"] Jul 11 23:37:04.807625 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:04.807578 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fba839d-05a8-40ba-97e5-812d0135ed91" path="/var/lib/kubelet/pods/6fba839d-05a8-40ba-97e5-812d0135ed91/volumes" Jul 11 23:37:10.059456 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:10.059424 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:37:10.059855 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:10.059487 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:37:10.080845 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:10.080822 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:37:10.502952 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:10.502927 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xh2gs" Jul 11 23:37:21.972318 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:21.972279 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:25.816537 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:25.816483 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:26.446624 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.446576 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:26.500038 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.500007 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb"] Jul 11 23:37:26.500350 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.500338 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fba839d-05a8-40ba-97e5-812d0135ed91" containerName="registry-server" Jul 11 23:37:26.500400 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.500351 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fba839d-05a8-40ba-97e5-812d0135ed91" containerName="registry-server" Jul 11 23:37:26.500441 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.500409 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fba839d-05a8-40ba-97e5-812d0135ed91" containerName="registry-server" Jul 11 23:37:26.504685 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.504667 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.507663 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.507642 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gwvd7\"" Jul 11 23:37:26.515030 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.515014 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb"] Jul 11 23:37:26.587818 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.587793 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9254f4bb-2773-46db-a340-2c465e0fd107-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.587924 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.587837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42rz\" (UniqueName: \"kubernetes.io/projected/9254f4bb-2773-46db-a340-2c465e0fd107-kube-api-access-q42rz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.688926 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.688902 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42rz\" (UniqueName: \"kubernetes.io/projected/9254f4bb-2773-46db-a340-2c465e0fd107-kube-api-access-q42rz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.689023 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.688955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9254f4bb-2773-46db-a340-2c465e0fd107-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.689280 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.689265 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9254f4bb-2773-46db-a340-2c465e0fd107-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.702879 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.702834 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42rz\" (UniqueName: \"kubernetes.io/projected/9254f4bb-2773-46db-a340-2c465e0fd107-kube-api-access-q42rz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.815102 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.815081 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:26.942041 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.942011 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb"] Jul 11 23:37:26.945451 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:37:26.945422 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9254f4bb_2773_46db_a340_2c465e0fd107.slice/crio-cfc47933e92cae6535863dd2319c6cdbba15b6bacea29fc3af426d7aee9a106c WatchSource:0}: Error finding container cfc47933e92cae6535863dd2319c6cdbba15b6bacea29fc3af426d7aee9a106c: Status 404 returned error can't find the container with id cfc47933e92cae6535863dd2319c6cdbba15b6bacea29fc3af426d7aee9a106c Jul 11 23:37:26.975473 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:26.975415 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:27.553634 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:27.553599 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" event={"ID":"9254f4bb-2773-46db-a340-2c465e0fd107","Type":"ContainerStarted","Data":"cfc47933e92cae6535863dd2319c6cdbba15b6bacea29fc3af426d7aee9a106c"} Jul 11 23:37:27.585178 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:27.585147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:29.197989 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.197955 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:29.551174 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.551106 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jclsm"] Jul 11 23:37:29.554215 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.554199 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" Jul 11 23:37:29.556866 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.556850 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pmx9p\"" Jul 11 23:37:29.564937 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.564915 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jclsm"] Jul 11 23:37:29.718065 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.718031 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcqs\" (UniqueName: \"kubernetes.io/projected/bd1007f2-e8ed-453c-92a2-36c5d0671354-kube-api-access-jwcqs\") pod \"authorino-operator-75cc8d5cb7-jclsm\" (UID: \"bd1007f2-e8ed-453c-92a2-36c5d0671354\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" Jul 11 23:37:29.819236 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.819155 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcqs\" (UniqueName: \"kubernetes.io/projected/bd1007f2-e8ed-453c-92a2-36c5d0671354-kube-api-access-jwcqs\") pod \"authorino-operator-75cc8d5cb7-jclsm\" (UID: \"bd1007f2-e8ed-453c-92a2-36c5d0671354\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" Jul 11 23:37:29.831656 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.831633 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcqs\" (UniqueName: \"kubernetes.io/projected/bd1007f2-e8ed-453c-92a2-36c5d0671354-kube-api-access-jwcqs\") pod \"authorino-operator-75cc8d5cb7-jclsm\" (UID: \"bd1007f2-e8ed-453c-92a2-36c5d0671354\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" Jul 11 23:37:29.866484 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:29.866448 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" Jul 11 23:37:30.020357 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:30.020285 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-jclsm"] Jul 11 23:37:30.024214 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:37:30.024134 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1007f2_e8ed_453c_92a2_36c5d0671354.slice/crio-4fa4a7f457f93fef828800f2c38abcbfd7c5abf21b590eef12caa2f177b872c2 WatchSource:0}: Error finding container 4fa4a7f457f93fef828800f2c38abcbfd7c5abf21b590eef12caa2f177b872c2: Status 404 returned error can't find the container with id 4fa4a7f457f93fef828800f2c38abcbfd7c5abf21b590eef12caa2f177b872c2 Jul 11 23:37:30.571459 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:30.571393 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" event={"ID":"bd1007f2-e8ed-453c-92a2-36c5d0671354","Type":"ContainerStarted","Data":"4fa4a7f457f93fef828800f2c38abcbfd7c5abf21b590eef12caa2f177b872c2"} Jul 11 23:37:30.592046 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:30.591965 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:30.799687 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:30.799579 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:32.987362 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:32.987334 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:33.089982 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:33.089954 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:33.588457 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:33.588417 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" event={"ID":"9254f4bb-2773-46db-a340-2c465e0fd107","Type":"ContainerStarted","Data":"2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8"} Jul 11 23:37:33.588762 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:33.588481 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:33.590004 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:33.589979 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" event={"ID":"bd1007f2-e8ed-453c-92a2-36c5d0671354","Type":"ContainerStarted","Data":"a62654a95441e49054403f0c866d19ca45a28d4209319f9a703539929d34689d"} Jul 11 23:37:33.590089 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:33.590029 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" Jul 11 23:37:33.611729 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:33.611685 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" podStartSLOduration=1.686362278 podStartE2EDuration="7.611673803s" totalImagesPullingTime="5.925311525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:33.609412495 +0000 UTC m=+389.265035749" watchObservedRunningTime="2026-07-11 23:37:33.611673803 +0000 UTC m=+389.267297056" Jul 11 23:37:33.625819 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:33.625777 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" podStartSLOduration=1.745280083 podStartE2EDuration="4.625766322s" totalImagesPullingTime="2.880486239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:33.62531563 +0000 UTC m=+389.280938882" watchObservedRunningTime="2026-07-11 23:37:33.625766322 +0000 UTC m=+389.281389576" Jul 11 23:37:34.583472 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.583436 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd"] Jul 11 23:37:34.587168 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.587151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:34.590144 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.590125 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mdlhd\"" Jul 11 23:37:34.596911 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.596888 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd"] Jul 11 23:37:34.763943 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.763903 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxt68\" (UniqueName: \"kubernetes.io/projected/c29e2776-0297-430d-b43c-564d32c03659-kube-api-access-pxt68\") pod \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" (UID: \"c29e2776-0297-430d-b43c-564d32c03659\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:34.865325 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.865243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt68\" (UniqueName: \"kubernetes.io/projected/c29e2776-0297-430d-b43c-564d32c03659-kube-api-access-pxt68\") pod \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" (UID: \"c29e2776-0297-430d-b43c-564d32c03659\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:34.873760 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.873731 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt68\" (UniqueName: \"kubernetes.io/projected/c29e2776-0297-430d-b43c-564d32c03659-kube-api-access-pxt68\") pod \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" (UID: \"c29e2776-0297-430d-b43c-564d32c03659\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:34.898423 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:34.898404 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:35.023059 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:35.023003 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd"] Jul 11 23:37:35.025309 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:37:35.025285 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc29e2776_0297_430d_b43c_564d32c03659.slice/crio-ad45f19700fe26d52f620b67955b496f0328cf8fb867a7550dd39641b7825750 WatchSource:0}: Error finding container ad45f19700fe26d52f620b67955b496f0328cf8fb867a7550dd39641b7825750: Status 404 returned error can't find the container with id ad45f19700fe26d52f620b67955b496f0328cf8fb867a7550dd39641b7825750 Jul 11 23:37:35.600425 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:35.600397 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" event={"ID":"c29e2776-0297-430d-b43c-564d32c03659","Type":"ContainerStarted","Data":"ad45f19700fe26d52f620b67955b496f0328cf8fb867a7550dd39641b7825750"} Jul 11 23:37:35.896460 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:35.896413 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:36.013433 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:36.013401 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:37.179012 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:37.178982 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:37.279351 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:37.279326 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:37.613305 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:37.613236 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" event={"ID":"c29e2776-0297-430d-b43c-564d32c03659","Type":"ContainerStarted","Data":"c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061"} Jul 11 23:37:37.613420 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:37.613363 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:37.634025 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:37.633981 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" podStartSLOduration=1.972096825 podStartE2EDuration="3.633969327s" totalImagesPullingTime="1.661872502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:37.632565198 +0000 UTC m=+393.288188452" watchObservedRunningTime="2026-07-11 23:37:37.633969327 +0000 UTC m=+393.289592579" Jul 11 23:37:43.392260 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:43.392229 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:43.538768 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:43.538734 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:43.638419 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:43.638385 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:44.134205 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:44.134165 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:44.234370 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:44.234344 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:44.597858 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:44.597833 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-jclsm" Jul 11 23:37:44.598330 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:44.597902 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.005343 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.005284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:46.106509 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.106483 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:46.270412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.270333 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb"] Jul 11 23:37:46.270567 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.270515 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" containerName="manager" containerID="cri-o://2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8" gracePeriod=2 Jul 11 23:37:46.282827 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.282800 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb"] Jul 11 23:37:46.283454 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.283419 2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"9254f4bb-2773-46db-a340-2c465e0fd107\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\": pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.290853 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.290828 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs"] Jul 11 23:37:46.291205 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.291192 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" containerName="manager" Jul 11 23:37:46.291264 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.291207 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" containerName="manager" Jul 11 23:37:46.291318 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.291305 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" containerName="manager" Jul 11 23:37:46.294234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.294219 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd"] Jul 11 23:37:46.294383 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.294363 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" podUID="c29e2776-0297-430d-b43c-564d32c03659" containerName="manager" containerID="cri-o://c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061" gracePeriod=2 Jul 11 23:37:46.294439 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.294389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.297100 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.297053 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.297216 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.297147 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.299291 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.299257 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.299405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.299325 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd"] Jul 11 23:37:46.301442 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.301416 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.303936 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.303911 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.305337 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.305318 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs"] Jul 11 23:37:46.319749 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.319729 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf"] Jul 11 23:37:46.320195 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.320172 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c29e2776-0297-430d-b43c-564d32c03659" containerName="manager" Jul 11 23:37:46.320195 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.320196 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c29e2776-0297-430d-b43c-564d32c03659" containerName="manager" Jul 11 23:37:46.320372 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.320288 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c29e2776-0297-430d-b43c-564d32c03659" containerName="manager" Jul 11 23:37:46.323094 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.323078 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" Jul 11 23:37:46.334072 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.334048 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf"] Jul 11 23:37:46.334600 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.334568 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.336847 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.336823 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.352537 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.349206 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6m4z\" (UniqueName: \"kubernetes.io/projected/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-kube-api-access-k6m4z\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sb5zs\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.352537 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.349341 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sb5zs\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.352537 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.349411 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxpm8\" (UniqueName: \"kubernetes.io/projected/f4c3cd5e-1449-4aae-88bf-234df0ed8e44-kube-api-access-sxpm8\") pod \"limitador-operator-controller-manager-859d7b78bd-kkqmf\" (UID: \"f4c3cd5e-1449-4aae-88bf-234df0ed8e44\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" Jul 11 23:37:46.450295 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.450255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sb5zs\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.450467 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.450315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxpm8\" (UniqueName: \"kubernetes.io/projected/f4c3cd5e-1449-4aae-88bf-234df0ed8e44-kube-api-access-sxpm8\") pod \"limitador-operator-controller-manager-859d7b78bd-kkqmf\" (UID: \"f4c3cd5e-1449-4aae-88bf-234df0ed8e44\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" Jul 11 23:37:46.450467 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.450343 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6m4z\" (UniqueName: \"kubernetes.io/projected/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-kube-api-access-k6m4z\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sb5zs\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.450770 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.450732 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sb5zs\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.462922 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.462897 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6m4z\" (UniqueName: \"kubernetes.io/projected/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-kube-api-access-k6m4z\") pod \"kuadrant-operator-controller-manager-8684c9ff49-sb5zs\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.463029 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.462985 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxpm8\" (UniqueName: \"kubernetes.io/projected/f4c3cd5e-1449-4aae-88bf-234df0ed8e44-kube-api-access-sxpm8\") pod \"limitador-operator-controller-manager-859d7b78bd-kkqmf\" (UID: \"f4c3cd5e-1449-4aae-88bf-234df0ed8e44\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" Jul 11 23:37:46.530470 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.530447 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.532474 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.532459 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.533238 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.533216 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.535786 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.535766 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.538056 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.538037 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.540278 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.540259 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.551138 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.551114 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9254f4bb-2773-46db-a340-2c465e0fd107-extensions-socket-volume\") pod \"9254f4bb-2773-46db-a340-2c465e0fd107\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " Jul 11 23:37:46.551229 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.551174 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q42rz\" (UniqueName: \"kubernetes.io/projected/9254f4bb-2773-46db-a340-2c465e0fd107-kube-api-access-q42rz\") pod \"9254f4bb-2773-46db-a340-2c465e0fd107\" (UID: \"9254f4bb-2773-46db-a340-2c465e0fd107\") " Jul 11 23:37:46.551229 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.551215 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxt68\" (UniqueName: \"kubernetes.io/projected/c29e2776-0297-430d-b43c-564d32c03659-kube-api-access-pxt68\") pod \"c29e2776-0297-430d-b43c-564d32c03659\" (UID: \"c29e2776-0297-430d-b43c-564d32c03659\") " Jul 11 23:37:46.551576 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.551555 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9254f4bb-2773-46db-a340-2c465e0fd107-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9254f4bb-2773-46db-a340-2c465e0fd107" (UID: "9254f4bb-2773-46db-a340-2c465e0fd107"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 23:37:46.553235 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.553213 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9254f4bb-2773-46db-a340-2c465e0fd107-kube-api-access-q42rz" (OuterVolumeSpecName: "kube-api-access-q42rz") pod "9254f4bb-2773-46db-a340-2c465e0fd107" (UID: "9254f4bb-2773-46db-a340-2c465e0fd107"). InnerVolumeSpecName "kube-api-access-q42rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:37:46.553312 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.553250 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29e2776-0297-430d-b43c-564d32c03659-kube-api-access-pxt68" (OuterVolumeSpecName: "kube-api-access-pxt68") pod "c29e2776-0297-430d-b43c-564d32c03659" (UID: "c29e2776-0297-430d-b43c-564d32c03659"). InnerVolumeSpecName "kube-api-access-pxt68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:37:46.651710 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.651686 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxt68\" (UniqueName: \"kubernetes.io/projected/c29e2776-0297-430d-b43c-564d32c03659-kube-api-access-pxt68\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:37:46.651710 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.651707 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9254f4bb-2773-46db-a340-2c465e0fd107-extensions-socket-volume\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:37:46.651844 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.651716 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42rz\" (UniqueName: \"kubernetes.io/projected/9254f4bb-2773-46db-a340-2c465e0fd107-kube-api-access-q42rz\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:37:46.656204 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.656181 2566 generic.go:356] "Generic (PLEG): container finished" podID="c29e2776-0297-430d-b43c-564d32c03659" containerID="c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061" exitCode=0 Jul 11 23:37:46.656323 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.656220 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.656323 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.656257 2566 scope.go:117] "RemoveContainer" containerID="c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061" Jul 11 23:37:46.657789 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.657766 2566 generic.go:356] "Generic (PLEG): container finished" podID="9254f4bb-2773-46db-a340-2c465e0fd107" containerID="2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8" exitCode=0 Jul 11 23:37:46.657897 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.657807 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.659134 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.659111 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.661218 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.661194 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.664531 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.664394 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.666585 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.666558 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.667268 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.667249 2566 scope.go:117] "RemoveContainer" containerID="c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061" Jul 11 23:37:46.667563 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.667542 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061\": container with ID starting with c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061 not found: ID does not exist" containerID="c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061" Jul 11 23:37:46.667663 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.667572 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061"} err="failed to get container status \"c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061\": rpc error: code = NotFound desc = could not find container \"c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061\": container with ID starting with c901a8dd397076000c570404039ed168745b365cfdab351b63ff3860450f1061 not found: ID does not exist" Jul 11 23:37:46.667663 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.667593 2566 scope.go:117] "RemoveContainer" containerID="2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8" Jul 11 23:37:46.671140 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.671118 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.673326 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.673301 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.675748 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.675647 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wzwxb\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wzwxb" Jul 11 23:37:46.675799 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.675779 2566 scope.go:117] "RemoveContainer" containerID="2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8" Jul 11 23:37:46.676010 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.675992 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8\": container with ID starting with 2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8 not found: ID does not exist" containerID="2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8" Jul 11 23:37:46.676057 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.676015 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8"} err="failed to get container status \"2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8\": rpc error: code = NotFound desc = could not find container \"2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8\": container with ID starting with 2553c7f767f545c2560f16cac80a10b9f029fd0f75e7c04d8aef0f5d89ab5af8 not found: ID does not exist" Jul 11 23:37:46.679626 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:37:46.679607 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-p7jzd\" is forbidden: User \"system:node:ip-10-0-137-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-251.ec2.internal' and this object" podUID="c29e2776-0297-430d-b43c-564d32c03659" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-p7jzd" Jul 11 23:37:46.712862 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.712840 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:46.722012 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.721997 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" Jul 11 23:37:46.809574 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.809548 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9254f4bb-2773-46db-a340-2c465e0fd107" path="/var/lib/kubelet/pods/9254f4bb-2773-46db-a340-2c465e0fd107/volumes" Jul 11 23:37:46.809877 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.809864 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c29e2776-0297-430d-b43c-564d32c03659" path="/var/lib/kubelet/pods/c29e2776-0297-430d-b43c-564d32c03659/volumes" Jul 11 23:37:46.858543 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.858515 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs"] Jul 11 23:37:46.861027 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:37:46.860994 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed34e6a_4eff_4975_9f2a_4763dcdaa461.slice/crio-ff5adc8389d015d246bdfe43a588b32fc0b63e24ee4e98933f7423cf49a962d7 WatchSource:0}: Error finding container ff5adc8389d015d246bdfe43a588b32fc0b63e24ee4e98933f7423cf49a962d7: Status 404 returned error can't find the container with id ff5adc8389d015d246bdfe43a588b32fc0b63e24ee4e98933f7423cf49a962d7 Jul 11 23:37:46.877743 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:46.877726 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf"] Jul 11 23:37:46.879622 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:37:46.879597 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c3cd5e_1449_4aae_88bf_234df0ed8e44.slice/crio-e5b3592cd054a9b6b333666770292c48f2e52dd1e9a9bc29c5bb031ad02afeb2 WatchSource:0}: Error finding container e5b3592cd054a9b6b333666770292c48f2e52dd1e9a9bc29c5bb031ad02afeb2: Status 404 returned error can't find the container with id e5b3592cd054a9b6b333666770292c48f2e52dd1e9a9bc29c5bb031ad02afeb2 Jul 11 23:37:47.200774 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.200745 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd"] Jul 11 23:37:47.204222 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.204202 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.216842 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.216822 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd"] Jul 11 23:37:47.254317 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.254289 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65693151-11a6-4c6c-ba2b-c8a94286ce00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9fztd\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.254420 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.254324 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7tf\" (UniqueName: \"kubernetes.io/projected/65693151-11a6-4c6c-ba2b-c8a94286ce00-kube-api-access-gv7tf\") pod \"kuadrant-operator-controller-manager-86878c769f-9fztd\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.355600 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.355562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65693151-11a6-4c6c-ba2b-c8a94286ce00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9fztd\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.355721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.355626 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7tf\" (UniqueName: \"kubernetes.io/projected/65693151-11a6-4c6c-ba2b-c8a94286ce00-kube-api-access-gv7tf\") pod \"kuadrant-operator-controller-manager-86878c769f-9fztd\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.355933 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.355913 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65693151-11a6-4c6c-ba2b-c8a94286ce00-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9fztd\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.365542 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.365495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7tf\" (UniqueName: \"kubernetes.io/projected/65693151-11a6-4c6c-ba2b-c8a94286ce00-kube-api-access-gv7tf\") pod \"kuadrant-operator-controller-manager-86878c769f-9fztd\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.514307 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.514232 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:47.664377 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.664341 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" event={"ID":"6ed34e6a-4eff-4975-9f2a-4763dcdaa461","Type":"ContainerStarted","Data":"1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb"} Jul 11 23:37:47.664377 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.664404 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" event={"ID":"6ed34e6a-4eff-4975-9f2a-4763dcdaa461","Type":"ContainerStarted","Data":"ff5adc8389d015d246bdfe43a588b32fc0b63e24ee4e98933f7423cf49a962d7"} Jul 11 23:37:47.666644 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.666621 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd"] Jul 11 23:37:47.668549 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:37:47.668524 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65693151_11a6_4c6c_ba2b_c8a94286ce00.slice/crio-eab1175973f1effe9e31b466242efb96080ddb21d204039546a8c05f70016f4d WatchSource:0}: Error finding container eab1175973f1effe9e31b466242efb96080ddb21d204039546a8c05f70016f4d: Status 404 returned error can't find the container with id eab1175973f1effe9e31b466242efb96080ddb21d204039546a8c05f70016f4d Jul 11 23:37:47.669026 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.669004 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" event={"ID":"f4c3cd5e-1449-4aae-88bf-234df0ed8e44","Type":"ContainerStarted","Data":"d2a3283d65bc97922ffea588f6f0310e022162f8ba99253d069a8275f5dd06ba"} Jul 11 23:37:47.669142 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.669050 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" event={"ID":"f4c3cd5e-1449-4aae-88bf-234df0ed8e44","Type":"ContainerStarted","Data":"e5b3592cd054a9b6b333666770292c48f2e52dd1e9a9bc29c5bb031ad02afeb2"} Jul 11 23:37:47.669142 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.669094 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" Jul 11 23:37:47.707880 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.707842 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" podStartSLOduration=1.707828497 podStartE2EDuration="1.707828497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:47.707174894 +0000 UTC m=+403.362798147" watchObservedRunningTime="2026-07-11 23:37:47.707828497 +0000 UTC m=+403.363451749" Jul 11 23:37:47.708977 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:47.708949 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" podStartSLOduration=1.708937159 podStartE2EDuration="1.708937159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:47.689230809 +0000 UTC m=+403.344854062" watchObservedRunningTime="2026-07-11 23:37:47.708937159 +0000 UTC m=+403.364560411" Jul 11 23:37:48.437015 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:48.436974 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:48.546840 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:48.546807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:48.675656 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:48.675627 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" event={"ID":"65693151-11a6-4c6c-ba2b-c8a94286ce00","Type":"ContainerStarted","Data":"2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b"} Jul 11 23:37:48.675838 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:48.675686 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" event={"ID":"65693151-11a6-4c6c-ba2b-c8a94286ce00","Type":"ContainerStarted","Data":"eab1175973f1effe9e31b466242efb96080ddb21d204039546a8c05f70016f4d"} Jul 11 23:37:48.676027 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:48.676008 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:37:48.676115 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:48.676048 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:37:48.697755 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:48.697682 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" podStartSLOduration=1.6976687689999999 podStartE2EDuration="1.697668769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:37:48.696854325 +0000 UTC m=+404.352477578" watchObservedRunningTime="2026-07-11 23:37:48.697668769 +0000 UTC m=+404.353292074" Jul 11 23:37:53.194817 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:53.194779 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:53.296246 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:53.296219 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:57.194319 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:57.194281 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:57.294245 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:57.294214 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:37:58.678212 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:58.678182 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-kkqmf" Jul 11 23:37:59.682000 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:37:59.681973 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:38:00.692300 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:00.692268 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:38:00.703561 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:00.703527 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:00.739850 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:00.739824 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs"] Jul 11 23:38:00.740056 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:00.740031 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" podUID="6ed34e6a-4eff-4975-9f2a-4763dcdaa461" containerName="manager" containerID="cri-o://1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb" gracePeriod=10 Jul 11 23:38:00.768668 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:00.768625 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:00.994999 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:00.994977 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:38:01.051297 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.051269 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6m4z\" (UniqueName: \"kubernetes.io/projected/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-kube-api-access-k6m4z\") pod \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " Jul 11 23:38:01.051410 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.051316 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-extensions-socket-volume\") pod \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\" (UID: \"6ed34e6a-4eff-4975-9f2a-4763dcdaa461\") " Jul 11 23:38:01.051710 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.051691 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6ed34e6a-4eff-4975-9f2a-4763dcdaa461" (UID: "6ed34e6a-4eff-4975-9f2a-4763dcdaa461"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 23:38:01.053202 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.053183 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-kube-api-access-k6m4z" (OuterVolumeSpecName: "kube-api-access-k6m4z") pod "6ed34e6a-4eff-4975-9f2a-4763dcdaa461" (UID: "6ed34e6a-4eff-4975-9f2a-4763dcdaa461"). InnerVolumeSpecName "kube-api-access-k6m4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:38:01.152041 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.152011 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6m4z\" (UniqueName: \"kubernetes.io/projected/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-kube-api-access-k6m4z\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:38:01.152159 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.152046 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ed34e6a-4eff-4975-9f2a-4763dcdaa461-extensions-socket-volume\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:38:01.743637 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.743609 2566 generic.go:356] "Generic (PLEG): container finished" podID="6ed34e6a-4eff-4975-9f2a-4763dcdaa461" containerID="1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb" exitCode=0 Jul 11 23:38:01.744044 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.743656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" event={"ID":"6ed34e6a-4eff-4975-9f2a-4763dcdaa461","Type":"ContainerDied","Data":"1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb"} Jul 11 23:38:01.744044 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.743658 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" Jul 11 23:38:01.744044 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.743697 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs" event={"ID":"6ed34e6a-4eff-4975-9f2a-4763dcdaa461","Type":"ContainerDied","Data":"ff5adc8389d015d246bdfe43a588b32fc0b63e24ee4e98933f7423cf49a962d7"} Jul 11 23:38:01.744044 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.743733 2566 scope.go:117] "RemoveContainer" containerID="1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb" Jul 11 23:38:01.752183 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.752166 2566 scope.go:117] "RemoveContainer" containerID="1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb" Jul 11 23:38:01.752439 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:38:01.752414 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb\": container with ID starting with 1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb not found: ID does not exist" containerID="1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb" Jul 11 23:38:01.752488 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.752439 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb"} err="failed to get container status \"1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb\": rpc error: code = NotFound desc = could not find container \"1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb\": container with ID starting with 1f916fa130c979cc0da84d9daf10a423d7851f3c4b35183615afd99275b0e8eb not found: ID does not exist" Jul 11 23:38:01.768778 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.768750 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs"] Jul 11 23:38:01.772723 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:01.772704 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-sb5zs"] Jul 11 23:38:02.811858 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:02.811826 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed34e6a-4eff-4975-9f2a-4763dcdaa461" path="/var/lib/kubelet/pods/6ed34e6a-4eff-4975-9f2a-4763dcdaa461/volumes" Jul 11 23:38:04.233689 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:04.233634 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:04.338123 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:04.338085 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:04.388032 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:04.387999 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:06.409294 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:06.409257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:06.508453 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:06.508419 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:08.072301 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:08.072264 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:08.118130 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:08.118102 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:08.850439 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:08.850409 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:08.899589 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:08.899547 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:10.032181 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:10.032147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:10.130620 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:10.130590 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:10.701428 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:10.701389 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:13.608310 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:13.608269 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:13.708886 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:13.708857 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:17.045830 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:17.045795 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:17.188791 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:17.188762 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:18.717285 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:18.717246 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:18.799986 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:18.799949 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:18.899211 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:18.899174 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:19.482066 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.482034 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9b54h"] Jul 11 23:38:19.482372 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.482360 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed34e6a-4eff-4975-9f2a-4763dcdaa461" containerName="manager" Jul 11 23:38:19.482427 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.482373 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed34e6a-4eff-4975-9f2a-4763dcdaa461" containerName="manager" Jul 11 23:38:19.482427 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.482423 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed34e6a-4eff-4975-9f2a-4763dcdaa461" containerName="manager" Jul 11 23:38:19.490295 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.490271 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9b54h" Jul 11 23:38:19.491064 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.491042 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9b54h"] Jul 11 23:38:19.492933 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.492912 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hcq9d\"" Jul 11 23:38:19.512468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.512439 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:19.569813 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.569776 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:19.587815 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.587794 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdc9z\" (UniqueName: \"kubernetes.io/projected/365279d7-8b07-41fd-b461-d76695f60e42-kube-api-access-pdc9z\") pod \"authorino-58cd955675-9b54h\" (UID: \"365279d7-8b07-41fd-b461-d76695f60e42\") " pod="kuadrant-system/authorino-58cd955675-9b54h" Jul 11 23:38:19.689099 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.689063 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc9z\" (UniqueName: \"kubernetes.io/projected/365279d7-8b07-41fd-b461-d76695f60e42-kube-api-access-pdc9z\") pod \"authorino-58cd955675-9b54h\" (UID: \"365279d7-8b07-41fd-b461-d76695f60e42\") " pod="kuadrant-system/authorino-58cd955675-9b54h" Jul 11 23:38:19.696944 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.696915 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc9z\" (UniqueName: \"kubernetes.io/projected/365279d7-8b07-41fd-b461-d76695f60e42-kube-api-access-pdc9z\") pod \"authorino-58cd955675-9b54h\" (UID: \"365279d7-8b07-41fd-b461-d76695f60e42\") " pod="kuadrant-system/authorino-58cd955675-9b54h" Jul 11 23:38:19.807187 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.807115 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9b54h" Jul 11 23:38:19.933658 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:19.933629 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9b54h"] Jul 11 23:38:19.937406 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:38:19.937381 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod365279d7_8b07_41fd_b461_d76695f60e42.slice/crio-462fb516de7d772f616c35b9b442c0ee56161a4184c59a29b3f2fbac4c967230 WatchSource:0}: Error finding container 462fb516de7d772f616c35b9b442c0ee56161a4184c59a29b3f2fbac4c967230: Status 404 returned error can't find the container with id 462fb516de7d772f616c35b9b442c0ee56161a4184c59a29b3f2fbac4c967230 Jul 11 23:38:20.836974 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:20.836917 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9b54h" event={"ID":"365279d7-8b07-41fd-b461-d76695f60e42","Type":"ContainerStarted","Data":"462fb516de7d772f616c35b9b442c0ee56161a4184c59a29b3f2fbac4c967230"} Jul 11 23:38:23.859440 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:23.859408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9b54h" event={"ID":"365279d7-8b07-41fd-b461-d76695f60e42","Type":"ContainerStarted","Data":"3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a"} Jul 11 23:38:23.872936 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:23.872891 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9b54h" podStartSLOduration=1.592918252 podStartE2EDuration="4.872874496s" totalImagesPullingTime="3.279956244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:38:23.872553856 +0000 UTC m=+439.528177109" watchObservedRunningTime="2026-07-11 23:38:23.872874496 +0000 UTC m=+439.528497753" Jul 11 23:38:27.287135 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.287060 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9b54h"] Jul 11 23:38:27.287521 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.287202 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9b54h" podUID="365279d7-8b07-41fd-b461-d76695f60e42" containerName="authorino" containerID="cri-o://3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a" gracePeriod=30 Jul 11 23:38:27.538579 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.538523 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9b54h" Jul 11 23:38:27.652848 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.652825 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdc9z\" (UniqueName: \"kubernetes.io/projected/365279d7-8b07-41fd-b461-d76695f60e42-kube-api-access-pdc9z\") pod \"365279d7-8b07-41fd-b461-d76695f60e42\" (UID: \"365279d7-8b07-41fd-b461-d76695f60e42\") " Jul 11 23:38:27.654757 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.654725 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365279d7-8b07-41fd-b461-d76695f60e42-kube-api-access-pdc9z" (OuterVolumeSpecName: "kube-api-access-pdc9z") pod "365279d7-8b07-41fd-b461-d76695f60e42" (UID: "365279d7-8b07-41fd-b461-d76695f60e42"). InnerVolumeSpecName "kube-api-access-pdc9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:38:27.753983 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.753941 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdc9z\" (UniqueName: \"kubernetes.io/projected/365279d7-8b07-41fd-b461-d76695f60e42-kube-api-access-pdc9z\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:38:27.880376 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.880291 2566 generic.go:356] "Generic (PLEG): container finished" podID="365279d7-8b07-41fd-b461-d76695f60e42" containerID="3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a" exitCode=0 Jul 11 23:38:27.880376 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.880329 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9b54h" Jul 11 23:38:27.880589 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.880381 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9b54h" event={"ID":"365279d7-8b07-41fd-b461-d76695f60e42","Type":"ContainerDied","Data":"3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a"} Jul 11 23:38:27.880589 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.880422 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9b54h" event={"ID":"365279d7-8b07-41fd-b461-d76695f60e42","Type":"ContainerDied","Data":"462fb516de7d772f616c35b9b442c0ee56161a4184c59a29b3f2fbac4c967230"} Jul 11 23:38:27.880589 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.880438 2566 scope.go:117] "RemoveContainer" containerID="3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a" Jul 11 23:38:27.893644 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.893620 2566 scope.go:117] "RemoveContainer" containerID="3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a" Jul 11 23:38:27.893940 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:38:27.893910 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a\": container with ID starting with 3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a not found: ID does not exist" containerID="3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a" Jul 11 23:38:27.894023 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.893945 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a"} err="failed to get container status \"3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a\": rpc error: code = NotFound desc = could not find container \"3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a\": container with ID starting with 3013d12a258be9ccddd4aeebf51348e54f8d2830ac0d062334758ca6d23cc72a not found: ID does not exist" Jul 11 23:38:27.906666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.906645 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9b54h"] Jul 11 23:38:27.909915 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:27.909896 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9b54h"] Jul 11 23:38:28.807421 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:28.807388 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="365279d7-8b07-41fd-b461-d76695f60e42" path="/var/lib/kubelet/pods/365279d7-8b07-41fd-b461-d76695f60e42/volumes" Jul 11 23:38:39.881334 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:39.881290 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:39.975373 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:39.975339 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:40.304890 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.304862 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9"] Jul 11 23:38:40.305418 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.305402 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365279d7-8b07-41fd-b461-d76695f60e42" containerName="authorino" Jul 11 23:38:40.305465 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.305422 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="365279d7-8b07-41fd-b461-d76695f60e42" containerName="authorino" Jul 11 23:38:40.305530 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.305518 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="365279d7-8b07-41fd-b461-d76695f60e42" containerName="authorino" Jul 11 23:38:40.310128 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.310113 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.313141 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.313106 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 11 23:38:40.313141 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.313119 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 11 23:38:40.314295 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.314276 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 11 23:38:40.314578 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.314276 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kw9hg\"" Jul 11 23:38:40.316057 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.316037 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9"] Jul 11 23:38:40.330437 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.330413 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:40.446279 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.446257 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ae682fd-6f3f-43df-9e20-4e014a9e757a-data\") pod \"postgres-69fb4594cf-mkmk9\" (UID: \"0ae682fd-6f3f-43df-9e20-4e014a9e757a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.446374 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.446309 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmh5\" (UniqueName: \"kubernetes.io/projected/0ae682fd-6f3f-43df-9e20-4e014a9e757a-kube-api-access-5kmh5\") pod \"postgres-69fb4594cf-mkmk9\" (UID: \"0ae682fd-6f3f-43df-9e20-4e014a9e757a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.547281 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.547257 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmh5\" (UniqueName: \"kubernetes.io/projected/0ae682fd-6f3f-43df-9e20-4e014a9e757a-kube-api-access-5kmh5\") pod \"postgres-69fb4594cf-mkmk9\" (UID: \"0ae682fd-6f3f-43df-9e20-4e014a9e757a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.547367 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.547307 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ae682fd-6f3f-43df-9e20-4e014a9e757a-data\") pod \"postgres-69fb4594cf-mkmk9\" (UID: \"0ae682fd-6f3f-43df-9e20-4e014a9e757a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.547609 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.547594 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ae682fd-6f3f-43df-9e20-4e014a9e757a-data\") pod \"postgres-69fb4594cf-mkmk9\" (UID: \"0ae682fd-6f3f-43df-9e20-4e014a9e757a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.555604 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.555556 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmh5\" (UniqueName: \"kubernetes.io/projected/0ae682fd-6f3f-43df-9e20-4e014a9e757a-kube-api-access-5kmh5\") pod \"postgres-69fb4594cf-mkmk9\" (UID: \"0ae682fd-6f3f-43df-9e20-4e014a9e757a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.621583 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.621559 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:40.753295 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.753272 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9"] Jul 11 23:38:40.755280 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:38:40.755253 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae682fd_6f3f_43df_9e20_4e014a9e757a.slice/crio-7e39975b1b518c64d4d6fc9cf1882244095d7efc9e0735c93f58601215596203 WatchSource:0}: Error finding container 7e39975b1b518c64d4d6fc9cf1882244095d7efc9e0735c93f58601215596203: Status 404 returned error can't find the container with id 7e39975b1b518c64d4d6fc9cf1882244095d7efc9e0735c93f58601215596203 Jul 11 23:38:40.946206 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:40.946180 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" event={"ID":"0ae682fd-6f3f-43df-9e20-4e014a9e757a","Type":"ContainerStarted","Data":"7e39975b1b518c64d4d6fc9cf1882244095d7efc9e0735c93f58601215596203"} Jul 11 23:38:45.981484 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:45.981444 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" event={"ID":"0ae682fd-6f3f-43df-9e20-4e014a9e757a","Type":"ContainerStarted","Data":"529e45574db0e723a2ad96689052f9cf4469e1eae8ab7f082df3a28747651543"} Jul 11 23:38:45.981848 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:45.981626 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:45.998476 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:45.998432 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" podStartSLOduration=1.353043642 podStartE2EDuration="5.998417675s" totalImagesPullingTime="4.645374033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:38:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:38:45.996048078 +0000 UTC m=+461.651671355" watchObservedRunningTime="2026-07-11 23:38:45.998417675 +0000 UTC m=+461.654040927" Jul 11 23:38:52.015752 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.015720 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mkmk9" Jul 11 23:38:52.305723 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.305633 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:52.433588 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.433556 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:52.528150 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.528120 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hl64n"] Jul 11 23:38:52.531704 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.531686 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hl64n" Jul 11 23:38:52.534259 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.534236 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hcq9d\"" Jul 11 23:38:52.537033 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.537013 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hl64n"] Jul 11 23:38:52.645515 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.645474 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdg2\" (UniqueName: \"kubernetes.io/projected/0a405f0e-32d5-49bb-8466-be15ed5148fc-kube-api-access-4xdg2\") pod \"authorino-5d4b67c446-hl64n\" (UID: \"0a405f0e-32d5-49bb-8466-be15ed5148fc\") " pod="kuadrant-system/authorino-5d4b67c446-hl64n" Jul 11 23:38:52.698025 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.698006 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hl64n"] Jul 11 23:38:52.698206 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:38:52.698190 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4xdg2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-hl64n" podUID="0a405f0e-32d5-49bb-8466-be15ed5148fc" Jul 11 23:38:52.719939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.719913 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-dszrf"] Jul 11 23:38:52.723165 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.723147 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:52.725784 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.725767 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 11 23:38:52.729598 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.729579 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-dszrf"] Jul 11 23:38:52.746711 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.746687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdg2\" (UniqueName: \"kubernetes.io/projected/0a405f0e-32d5-49bb-8466-be15ed5148fc-kube-api-access-4xdg2\") pod \"authorino-5d4b67c446-hl64n\" (UID: \"0a405f0e-32d5-49bb-8466-be15ed5148fc\") " pod="kuadrant-system/authorino-5d4b67c446-hl64n" Jul 11 23:38:52.757229 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.757211 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdg2\" (UniqueName: \"kubernetes.io/projected/0a405f0e-32d5-49bb-8466-be15ed5148fc-kube-api-access-4xdg2\") pod \"authorino-5d4b67c446-hl64n\" (UID: \"0a405f0e-32d5-49bb-8466-be15ed5148fc\") " pod="kuadrant-system/authorino-5d4b67c446-hl64n" Jul 11 23:38:52.760485 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.760466 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-dszrf"] Jul 11 23:38:52.760668 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:38:52.760651 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-d42z5 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-d42z5 tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-dszrf" podUID="3d1ff618-2bb0-4457-b606-8160929feef6" Jul 11 23:38:52.791894 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.791871 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-667d7b7f75-b2ql2"] Jul 11 23:38:52.795118 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.795102 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:52.801425 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.801405 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-667d7b7f75-b2ql2"] Jul 11 23:38:52.848012 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.847988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d1ff618-2bb0-4457-b606-8160929feef6-tls-cert\") pod \"authorino-76877994-dszrf\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:52.848105 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.848021 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d42z5\" (UniqueName: \"kubernetes.io/projected/3d1ff618-2bb0-4457-b606-8160929feef6-kube-api-access-d42z5\") pod \"authorino-76877994-dszrf\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:52.948637 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.948579 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8305899-7487-4329-8f1d-f9424b3c57e4-tls-cert\") pod \"authorino-667d7b7f75-b2ql2\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:52.948637 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.948617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d42z5\" (UniqueName: \"kubernetes.io/projected/3d1ff618-2bb0-4457-b606-8160929feef6-kube-api-access-d42z5\") pod \"authorino-76877994-dszrf\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:52.948788 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.948659 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lmdj\" (UniqueName: \"kubernetes.io/projected/a8305899-7487-4329-8f1d-f9424b3c57e4-kube-api-access-4lmdj\") pod \"authorino-667d7b7f75-b2ql2\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:52.948788 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.948756 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d1ff618-2bb0-4457-b606-8160929feef6-tls-cert\") pod \"authorino-76877994-dszrf\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:52.951228 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.951209 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d1ff618-2bb0-4457-b606-8160929feef6-tls-cert\") pod \"authorino-76877994-dszrf\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:52.955686 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:52.955670 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d42z5\" (UniqueName: \"kubernetes.io/projected/3d1ff618-2bb0-4457-b606-8160929feef6-kube-api-access-d42z5\") pod \"authorino-76877994-dszrf\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:53.012669 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.012649 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hl64n" Jul 11 23:38:53.012748 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.012649 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:53.030132 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.030114 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hl64n" Jul 11 23:38:53.033375 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.033359 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:53.049976 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.049957 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8305899-7487-4329-8f1d-f9424b3c57e4-tls-cert\") pod \"authorino-667d7b7f75-b2ql2\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:53.050068 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.049994 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lmdj\" (UniqueName: \"kubernetes.io/projected/a8305899-7487-4329-8f1d-f9424b3c57e4-kube-api-access-4lmdj\") pod \"authorino-667d7b7f75-b2ql2\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:53.052100 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.052078 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8305899-7487-4329-8f1d-f9424b3c57e4-tls-cert\") pod \"authorino-667d7b7f75-b2ql2\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:53.056879 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.056859 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lmdj\" (UniqueName: \"kubernetes.io/projected/a8305899-7487-4329-8f1d-f9424b3c57e4-kube-api-access-4lmdj\") pod \"authorino-667d7b7f75-b2ql2\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:53.110868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.110847 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:38:53.150590 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.150567 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d42z5\" (UniqueName: \"kubernetes.io/projected/3d1ff618-2bb0-4457-b606-8160929feef6-kube-api-access-d42z5\") pod \"3d1ff618-2bb0-4457-b606-8160929feef6\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " Jul 11 23:38:53.150728 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.150619 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d1ff618-2bb0-4457-b606-8160929feef6-tls-cert\") pod \"3d1ff618-2bb0-4457-b606-8160929feef6\" (UID: \"3d1ff618-2bb0-4457-b606-8160929feef6\") " Jul 11 23:38:53.150728 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.150665 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xdg2\" (UniqueName: \"kubernetes.io/projected/0a405f0e-32d5-49bb-8466-be15ed5148fc-kube-api-access-4xdg2\") pod \"0a405f0e-32d5-49bb-8466-be15ed5148fc\" (UID: \"0a405f0e-32d5-49bb-8466-be15ed5148fc\") " Jul 11 23:38:53.152469 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.152441 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1ff618-2bb0-4457-b606-8160929feef6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3d1ff618-2bb0-4457-b606-8160929feef6" (UID: "3d1ff618-2bb0-4457-b606-8160929feef6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:38:53.152574 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.152486 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1ff618-2bb0-4457-b606-8160929feef6-kube-api-access-d42z5" (OuterVolumeSpecName: "kube-api-access-d42z5") pod "3d1ff618-2bb0-4457-b606-8160929feef6" (UID: "3d1ff618-2bb0-4457-b606-8160929feef6"). InnerVolumeSpecName "kube-api-access-d42z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:38:53.152627 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.152582 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a405f0e-32d5-49bb-8466-be15ed5148fc-kube-api-access-4xdg2" (OuterVolumeSpecName: "kube-api-access-4xdg2") pod "0a405f0e-32d5-49bb-8466-be15ed5148fc" (UID: "0a405f0e-32d5-49bb-8466-be15ed5148fc"). InnerVolumeSpecName "kube-api-access-4xdg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:38:53.235539 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.235513 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-667d7b7f75-b2ql2"] Jul 11 23:38:53.237680 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:38:53.237647 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8305899_7487_4329_8f1d_f9424b3c57e4.slice/crio-4283ff9f1fb1d229a5248496d9128730efe744971f2abd83e27f154168b68a5a WatchSource:0}: Error finding container 4283ff9f1fb1d229a5248496d9128730efe744971f2abd83e27f154168b68a5a: Status 404 returned error can't find the container with id 4283ff9f1fb1d229a5248496d9128730efe744971f2abd83e27f154168b68a5a Jul 11 23:38:53.251859 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.251836 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xdg2\" (UniqueName: \"kubernetes.io/projected/0a405f0e-32d5-49bb-8466-be15ed5148fc-kube-api-access-4xdg2\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:38:53.251859 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.251861 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d42z5\" (UniqueName: \"kubernetes.io/projected/3d1ff618-2bb0-4457-b606-8160929feef6-kube-api-access-d42z5\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:38:53.251975 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:53.251871 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d1ff618-2bb0-4457-b606-8160929feef6-tls-cert\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:38:54.018417 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.018382 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" event={"ID":"a8305899-7487-4329-8f1d-f9424b3c57e4","Type":"ContainerStarted","Data":"94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac"} Jul 11 23:38:54.018417 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.018424 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" event={"ID":"a8305899-7487-4329-8f1d-f9424b3c57e4","Type":"ContainerStarted","Data":"4283ff9f1fb1d229a5248496d9128730efe744971f2abd83e27f154168b68a5a"} Jul 11 23:38:54.018712 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.018544 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-hl64n" Jul 11 23:38:54.018712 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.018619 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-dszrf" Jul 11 23:38:54.033616 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.033575 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" podStartSLOduration=1.701612715 podStartE2EDuration="2.033562504s" totalImagesPullingTime="331.949789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:38:54.032530174 +0000 UTC m=+469.688153433" watchObservedRunningTime="2026-07-11 23:38:54.033562504 +0000 UTC m=+469.689185757" Jul 11 23:38:54.055866 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.055835 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hl64n"] Jul 11 23:38:54.065366 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.064694 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-hl64n"] Jul 11 23:38:54.091241 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.091216 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-dszrf"] Jul 11 23:38:54.094317 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.094297 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-dszrf"] Jul 11 23:38:54.807608 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.807578 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a405f0e-32d5-49bb-8466-be15ed5148fc" path="/var/lib/kubelet/pods/0a405f0e-32d5-49bb-8466-be15ed5148fc/volumes" Jul 11 23:38:54.807818 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:54.807807 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d1ff618-2bb0-4457-b606-8160929feef6" path="/var/lib/kubelet/pods/3d1ff618-2bb0-4457-b606-8160929feef6/volumes" Jul 11 23:38:58.575944 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:58.575912 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:58.673657 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:58.673626 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:58.742443 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:58.742413 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:38:58.797929 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:38:58.797904 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:01.946653 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:01.946614 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:02.039437 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:02.039404 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:33.566083 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:33.566046 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:33.658593 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:33.658560 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:34.027600 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:34.027546 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:34.113274 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:34.113242 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:34.449262 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:34.449223 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:34.547355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:34.547318 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:44.644324 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:44.644285 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:44.748615 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:44.748580 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:39:53.240945 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:39:53.240909 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:40:03.422801 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:03.422725 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:40:15.713317 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.713278 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc"] Jul 11 23:40:15.716954 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.716932 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.719662 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.719645 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 11 23:40:15.721398 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.721373 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 11 23:40:15.721475 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.721382 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kq42x\"" Jul 11 23:40:15.721475 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.721434 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 11 23:40:15.726678 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.726656 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc"] Jul 11 23:40:15.820048 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.820017 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:40:15.884243 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.884210 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.884243 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.884242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.884404 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.884259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.884404 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.884382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwzb\" (UniqueName: \"kubernetes.io/projected/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-kube-api-access-whwzb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.884489 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.884418 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.884489 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.884460 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.985799 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.985735 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.985799 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.985772 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.985799 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.985789 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.986049 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.985808 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.986049 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.985852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwzb\" (UniqueName: \"kubernetes.io/projected/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-kube-api-access-whwzb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.986049 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.985867 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.986202 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.986170 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.986249 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.986227 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.986324 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.986298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.988023 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.988007 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.988251 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.988233 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:15.993424 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:15.993404 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwzb\" (UniqueName: \"kubernetes.io/projected/aaa1332a-bea2-46b6-9df4-caa60d58f8e8-kube-api-access-whwzb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s99pc\" (UID: \"aaa1332a-bea2-46b6-9df4-caa60d58f8e8\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:16.028558 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:16.028536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:16.157196 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:16.157174 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc"] Jul 11 23:40:16.159653 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:40:16.159622 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa1332a_bea2_46b6_9df4_caa60d58f8e8.slice/crio-ecae1998a0d0af32ff13c54a64fc3a43fe2b473120b76bea4cb5583a519182a2 WatchSource:0}: Error finding container ecae1998a0d0af32ff13c54a64fc3a43fe2b473120b76bea4cb5583a519182a2: Status 404 returned error can't find the container with id ecae1998a0d0af32ff13c54a64fc3a43fe2b473120b76bea4cb5583a519182a2 Jul 11 23:40:16.387554 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:16.387463 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" event={"ID":"aaa1332a-bea2-46b6-9df4-caa60d58f8e8","Type":"ContainerStarted","Data":"ecae1998a0d0af32ff13c54a64fc3a43fe2b473120b76bea4cb5583a519182a2"} Jul 11 23:40:18.398353 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:18.398315 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" event={"ID":"aaa1332a-bea2-46b6-9df4-caa60d58f8e8","Type":"ContainerStarted","Data":"56650468f5e2410b638c9ad78548343f2f4295676ac7cfbe621d619c4eab3ed0"} Jul 11 23:40:18.415732 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:18.415683 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" podStartSLOduration=2.225796787 podStartE2EDuration="3.415668162s" totalImagesPullingTime="1.189871375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:40:18.413689742 +0000 UTC m=+554.069312995" watchObservedRunningTime="2026-07-11 23:40:18.415668162 +0000 UTC m=+554.071291414" Jul 11 23:40:25.245272 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:25.245237 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:40:26.028837 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:26.028809 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:26.029040 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:26.028904 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:26.041342 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:26.041321 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:26.444974 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:26.444944 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s99pc" Jul 11 23:40:34.010959 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.010929 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s"] Jul 11 23:40:34.014739 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.014716 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.017647 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.017620 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 11 23:40:34.026092 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.026072 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s"] Jul 11 23:40:34.047496 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.047461 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:40:34.132155 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.132128 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6590983-dbf5-4888-b2d3-d135540ff2d2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.132286 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.132165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.132286 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.132253 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5vj\" (UniqueName: \"kubernetes.io/projected/e6590983-dbf5-4888-b2d3-d135540ff2d2-kube-api-access-hp5vj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.132375 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.132290 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.132375 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.132310 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.132375 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.132368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.232796 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.232775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.232904 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.232803 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.232904 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.232837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.232904 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.232875 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6590983-dbf5-4888-b2d3-d135540ff2d2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.233019 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.232909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.233164 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.233125 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5vj\" (UniqueName: \"kubernetes.io/projected/e6590983-dbf5-4888-b2d3-d135540ff2d2-kube-api-access-hp5vj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.233235 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.233201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.233289 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.233268 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.233393 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.233371 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.235089 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.235067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6590983-dbf5-4888-b2d3-d135540ff2d2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.235246 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.235229 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6590983-dbf5-4888-b2d3-d135540ff2d2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.241359 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.241336 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5vj\" (UniqueName: \"kubernetes.io/projected/e6590983-dbf5-4888-b2d3-d135540ff2d2-kube-api-access-hp5vj\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-nqq9s\" (UID: \"e6590983-dbf5-4888-b2d3-d135540ff2d2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.328209 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.328152 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:34.463595 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.463571 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s"] Jul 11 23:40:34.465170 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:40:34.465133 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6590983_dbf5_4888_b2d3_d135540ff2d2.slice/crio-010c6b266f6aec8bfce937284091b78493ccd23b49a3683f2fef238ce765f454 WatchSource:0}: Error finding container 010c6b266f6aec8bfce937284091b78493ccd23b49a3683f2fef238ce765f454: Status 404 returned error can't find the container with id 010c6b266f6aec8bfce937284091b78493ccd23b49a3683f2fef238ce765f454 Jul 11 23:40:34.473947 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:34.473922 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" event={"ID":"e6590983-dbf5-4888-b2d3-d135540ff2d2","Type":"ContainerStarted","Data":"010c6b266f6aec8bfce937284091b78493ccd23b49a3683f2fef238ce765f454"} Jul 11 23:40:35.480076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:35.480036 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" event={"ID":"e6590983-dbf5-4888-b2d3-d135540ff2d2","Type":"ContainerStarted","Data":"142b9a1ce959c749898812d2a4ffbfaf05dc39a204890914ad74b9c184d0ac6f"} Jul 11 23:40:35.497970 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:35.497927 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" podStartSLOduration=2.333357858 podStartE2EDuration="2.497912476s" totalImagesPullingTime="164.554618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:40:35.497068091 +0000 UTC m=+571.152691343" watchObservedRunningTime="2026-07-11 23:40:35.497912476 +0000 UTC m=+571.153535729" Jul 11 23:40:39.612826 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.612794 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz"] Jul 11 23:40:39.619785 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.619765 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.622989 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.622966 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 11 23:40:39.623522 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.623486 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz"] Jul 11 23:40:39.638810 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.638788 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:40:39.676628 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.676601 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.676740 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.676637 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzq26\" (UniqueName: \"kubernetes.io/projected/1b8da3b6-2929-4133-857b-b887548a8d52-kube-api-access-wzq26\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.676740 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.676662 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.676740 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.676686 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.676900 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.676781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8da3b6-2929-4133-857b-b887548a8d52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.676900 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.676839 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778063 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778036 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778190 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzq26\" (UniqueName: \"kubernetes.io/projected/1b8da3b6-2929-4133-857b-b887548a8d52-kube-api-access-wzq26\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778190 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778283 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778283 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8da3b6-2929-4133-857b-b887548a8d52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778380 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778472 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778452 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778556 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778475 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.778625 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.778598 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.780351 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.780331 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b8da3b6-2929-4133-857b-b887548a8d52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.780608 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.780592 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b8da3b6-2929-4133-857b-b887548a8d52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.788044 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.788020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzq26\" (UniqueName: \"kubernetes.io/projected/1b8da3b6-2929-4133-857b-b887548a8d52-kube-api-access-wzq26\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz\" (UID: \"1b8da3b6-2929-4133-857b-b887548a8d52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:39.930080 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:39.930055 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:40.057222 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:40.057196 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz"] Jul 11 23:40:40.058836 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:40:40.058814 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b8da3b6_2929_4133_857b_b887548a8d52.slice/crio-a06201f80bb9a54c8acb7b749848fc04b9628913d7934d6d10e7d8382f17c381 WatchSource:0}: Error finding container a06201f80bb9a54c8acb7b749848fc04b9628913d7934d6d10e7d8382f17c381: Status 404 returned error can't find the container with id a06201f80bb9a54c8acb7b749848fc04b9628913d7934d6d10e7d8382f17c381 Jul 11 23:40:40.505783 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:40.505711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" event={"ID":"1b8da3b6-2929-4133-857b-b887548a8d52","Type":"ContainerStarted","Data":"4b67f30e9b2fe4fca8a474a982d2bcd876ffcd72074912e91299f19f33f55630"} Jul 11 23:40:40.505783 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:40.505755 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" event={"ID":"1b8da3b6-2929-4133-857b-b887548a8d52","Type":"ContainerStarted","Data":"a06201f80bb9a54c8acb7b749848fc04b9628913d7934d6d10e7d8382f17c381"} Jul 11 23:40:40.524513 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:40.524453 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" podStartSLOduration=1.345060457 podStartE2EDuration="1.524437091s" totalImagesPullingTime="179.376634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:40:40.523300173 +0000 UTC m=+576.178923426" watchObservedRunningTime="2026-07-11 23:40:40.524437091 +0000 UTC m=+576.180060344" Jul 11 23:40:44.328945 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:44.328908 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:44.329412 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:44.329001 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:44.346964 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:44.346944 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:44.537483 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:44.537454 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-nqq9s" Jul 11 23:40:49.931214 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:49.931183 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:49.931628 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:49.931226 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:49.943834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:49.943814 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:40:50.564994 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:40:50.564970 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz" Jul 11 23:41:04.709452 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:04.709419 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:41:04.711811 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:04.711784 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:41:04.713801 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:04.713783 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:41:04.715973 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:04.715957 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:41:23.253919 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:23.253874 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:41:48.156969 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:48.156930 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:41:48.225288 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:48.225259 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:41:49.259834 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.259799 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:41:49.350315 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.350259 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:41:49.391819 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.391786 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-667d7b7f75-b2ql2"] Jul 11 23:41:49.392588 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.392004 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" podUID="a8305899-7487-4329-8f1d-f9424b3c57e4" containerName="authorino" containerID="cri-o://94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac" gracePeriod=30 Jul 11 23:41:49.645792 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.645770 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:41:49.729651 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.729623 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lmdj\" (UniqueName: \"kubernetes.io/projected/a8305899-7487-4329-8f1d-f9424b3c57e4-kube-api-access-4lmdj\") pod \"a8305899-7487-4329-8f1d-f9424b3c57e4\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " Jul 11 23:41:49.729813 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.729657 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8305899-7487-4329-8f1d-f9424b3c57e4-tls-cert\") pod \"a8305899-7487-4329-8f1d-f9424b3c57e4\" (UID: \"a8305899-7487-4329-8f1d-f9424b3c57e4\") " Jul 11 23:41:49.731721 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.731677 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8305899-7487-4329-8f1d-f9424b3c57e4-kube-api-access-4lmdj" (OuterVolumeSpecName: "kube-api-access-4lmdj") pod "a8305899-7487-4329-8f1d-f9424b3c57e4" (UID: "a8305899-7487-4329-8f1d-f9424b3c57e4"). InnerVolumeSpecName "kube-api-access-4lmdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:41:49.739770 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.739736 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8305899-7487-4329-8f1d-f9424b3c57e4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a8305899-7487-4329-8f1d-f9424b3c57e4" (UID: "a8305899-7487-4329-8f1d-f9424b3c57e4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 11 23:41:49.828983 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.828902 2566 generic.go:356] "Generic (PLEG): container finished" podID="a8305899-7487-4329-8f1d-f9424b3c57e4" containerID="94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac" exitCode=0 Jul 11 23:41:49.828983 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.828941 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" Jul 11 23:41:49.829161 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.828984 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" event={"ID":"a8305899-7487-4329-8f1d-f9424b3c57e4","Type":"ContainerDied","Data":"94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac"} Jul 11 23:41:49.829161 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.829023 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-667d7b7f75-b2ql2" event={"ID":"a8305899-7487-4329-8f1d-f9424b3c57e4","Type":"ContainerDied","Data":"4283ff9f1fb1d229a5248496d9128730efe744971f2abd83e27f154168b68a5a"} Jul 11 23:41:49.829161 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.829046 2566 scope.go:117] "RemoveContainer" containerID="94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac" Jul 11 23:41:49.830242 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.830225 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lmdj\" (UniqueName: \"kubernetes.io/projected/a8305899-7487-4329-8f1d-f9424b3c57e4-kube-api-access-4lmdj\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:41:49.830317 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.830247 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a8305899-7487-4329-8f1d-f9424b3c57e4-tls-cert\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:41:49.837740 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.837723 2566 scope.go:117] "RemoveContainer" containerID="94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac" Jul 11 23:41:49.837988 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:41:49.837973 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac\": container with ID starting with 94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac not found: ID does not exist" containerID="94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac" Jul 11 23:41:49.838036 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.837995 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac"} err="failed to get container status \"94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac\": rpc error: code = NotFound desc = could not find container \"94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac\": container with ID starting with 94a627f7bc4737a25a3f5367fae538db7d918d590352c99bf1d8d935763540ac not found: ID does not exist" Jul 11 23:41:49.849932 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.849914 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-667d7b7f75-b2ql2"] Jul 11 23:41:49.851522 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:49.851488 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-667d7b7f75-b2ql2"] Jul 11 23:41:50.806885 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:41:50.806853 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8305899-7487-4329-8f1d-f9424b3c57e4" path="/var/lib/kubelet/pods/a8305899-7487-4329-8f1d-f9424b3c57e4/volumes" Jul 11 23:42:05.700101 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:42:05.700071 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:42:05.750875 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:42:05.750847 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:42:22.117231 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:42:22.117192 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:42:59.530369 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:42:59.530274 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:00.928309 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:00.928280 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:07.551815 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:07.551779 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:07.600165 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:07.600131 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:41.363046 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:41.363015 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:41.453860 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:41.453824 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:41.575690 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:41.575661 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:47.212740 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:47.212704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:47.315345 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:47.315314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:47.412939 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:47.412905 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:47.588094 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:47.588004 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:53.129042 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:53.129009 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:43:53.230578 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:43:53.230544 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:44:40.776057 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:44:40.776020 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:46:04.736405 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:46:04.736330 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:46:04.740652 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:46:04.740630 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:46:04.741111 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:46:04.741090 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:46:04.744655 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:46:04.744623 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:47:30.594113 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:30.594030 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:47:43.180614 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:43.180581 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd"] Jul 11 23:47:43.181163 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:43.180816 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" podUID="65693151-11a6-4c6c-ba2b-c8a94286ce00" containerName="manager" containerID="cri-o://2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b" gracePeriod=10 Jul 11 23:47:46.043341 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.043315 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:47:46.151249 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.151228 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65693151-11a6-4c6c-ba2b-c8a94286ce00-extensions-socket-volume\") pod \"65693151-11a6-4c6c-ba2b-c8a94286ce00\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " Jul 11 23:47:46.151422 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.151271 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv7tf\" (UniqueName: \"kubernetes.io/projected/65693151-11a6-4c6c-ba2b-c8a94286ce00-kube-api-access-gv7tf\") pod \"65693151-11a6-4c6c-ba2b-c8a94286ce00\" (UID: \"65693151-11a6-4c6c-ba2b-c8a94286ce00\") " Jul 11 23:47:46.151666 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.151638 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65693151-11a6-4c6c-ba2b-c8a94286ce00-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "65693151-11a6-4c6c-ba2b-c8a94286ce00" (UID: "65693151-11a6-4c6c-ba2b-c8a94286ce00"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 11 23:47:46.153317 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.153295 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65693151-11a6-4c6c-ba2b-c8a94286ce00-kube-api-access-gv7tf" (OuterVolumeSpecName: "kube-api-access-gv7tf") pod "65693151-11a6-4c6c-ba2b-c8a94286ce00" (UID: "65693151-11a6-4c6c-ba2b-c8a94286ce00"). InnerVolumeSpecName "kube-api-access-gv7tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 11 23:47:46.252727 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.252697 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7tf\" (UniqueName: \"kubernetes.io/projected/65693151-11a6-4c6c-ba2b-c8a94286ce00-kube-api-access-gv7tf\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:47:46.252727 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.252721 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/65693151-11a6-4c6c-ba2b-c8a94286ce00-extensions-socket-volume\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 11 23:47:46.488003 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.487917 2566 generic.go:356] "Generic (PLEG): container finished" podID="65693151-11a6-4c6c-ba2b-c8a94286ce00" containerID="2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b" exitCode=0 Jul 11 23:47:46.488003 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.487970 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" Jul 11 23:47:46.488003 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.487985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" event={"ID":"65693151-11a6-4c6c-ba2b-c8a94286ce00","Type":"ContainerDied","Data":"2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b"} Jul 11 23:47:46.488234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.488036 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd" event={"ID":"65693151-11a6-4c6c-ba2b-c8a94286ce00","Type":"ContainerDied","Data":"eab1175973f1effe9e31b466242efb96080ddb21d204039546a8c05f70016f4d"} Jul 11 23:47:46.488234 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.488066 2566 scope.go:117] "RemoveContainer" containerID="2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b" Jul 11 23:47:46.497754 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.497734 2566 scope.go:117] "RemoveContainer" containerID="2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b" Jul 11 23:47:46.498027 ip-10-0-137-251 kubenswrapper[2566]: E0711 23:47:46.498011 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b\": container with ID starting with 2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b not found: ID does not exist" containerID="2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b" Jul 11 23:47:46.498091 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.498039 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b"} err="failed to get container status \"2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b\": rpc error: code = NotFound desc = could not find container \"2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b\": container with ID starting with 2319c2c1b9e65328aaaf713eeed7237b8f2d9c914122d6edb768aed76d71072b not found: ID does not exist" Jul 11 23:47:46.508734 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.508673 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd"] Jul 11 23:47:46.510912 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.510892 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9fztd"] Jul 11 23:47:46.807101 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:47:46.807032 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65693151-11a6-4c6c-ba2b-c8a94286ce00" path="/var/lib/kubelet/pods/65693151-11a6-4c6c-ba2b-c8a94286ce00/volumes" Jul 11 23:48:21.568684 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:21.568633 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:48:44.281477 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.281447 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f"] Jul 11 23:48:44.281868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.281807 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8305899-7487-4329-8f1d-f9424b3c57e4" containerName="authorino" Jul 11 23:48:44.281868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.281817 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8305899-7487-4329-8f1d-f9424b3c57e4" containerName="authorino" Jul 11 23:48:44.281868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.281832 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65693151-11a6-4c6c-ba2b-c8a94286ce00" containerName="manager" Jul 11 23:48:44.281868 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.281837 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65693151-11a6-4c6c-ba2b-c8a94286ce00" containerName="manager" Jul 11 23:48:44.282002 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.281913 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65693151-11a6-4c6c-ba2b-c8a94286ce00" containerName="manager" Jul 11 23:48:44.282002 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.281924 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8305899-7487-4329-8f1d-f9424b3c57e4" containerName="authorino" Jul 11 23:48:44.284942 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.284927 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.287998 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.287979 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gwvd7\"" Jul 11 23:48:44.295923 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.295904 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f"] Jul 11 23:48:44.407441 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.407414 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e054f7-67f1-4acf-a65a-0569e16f8105-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vpv4f\" (UID: \"45e054f7-67f1-4acf-a65a-0569e16f8105\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.407589 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.407448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrl4z\" (UniqueName: \"kubernetes.io/projected/45e054f7-67f1-4acf-a65a-0569e16f8105-kube-api-access-rrl4z\") pod \"kuadrant-operator-controller-manager-86878c769f-vpv4f\" (UID: \"45e054f7-67f1-4acf-a65a-0569e16f8105\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.508689 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.508659 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e054f7-67f1-4acf-a65a-0569e16f8105-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vpv4f\" (UID: \"45e054f7-67f1-4acf-a65a-0569e16f8105\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.508809 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.508696 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrl4z\" (UniqueName: \"kubernetes.io/projected/45e054f7-67f1-4acf-a65a-0569e16f8105-kube-api-access-rrl4z\") pod \"kuadrant-operator-controller-manager-86878c769f-vpv4f\" (UID: \"45e054f7-67f1-4acf-a65a-0569e16f8105\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.509012 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.508993 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45e054f7-67f1-4acf-a65a-0569e16f8105-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-vpv4f\" (UID: \"45e054f7-67f1-4acf-a65a-0569e16f8105\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.520010 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.519991 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrl4z\" (UniqueName: \"kubernetes.io/projected/45e054f7-67f1-4acf-a65a-0569e16f8105-kube-api-access-rrl4z\") pod \"kuadrant-operator-controller-manager-86878c769f-vpv4f\" (UID: \"45e054f7-67f1-4acf-a65a-0569e16f8105\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.595737 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.595679 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:44.745823 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.745790 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f"] Jul 11 23:48:44.746926 ip-10-0-137-251 kubenswrapper[2566]: W0711 23:48:44.746885 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e054f7_67f1_4acf_a65a_0569e16f8105.slice/crio-ce61e1524a8c7df6e5c170b8ee42f656f814f5b5dfe3d29c98d3533bacbd9b26 WatchSource:0}: Error finding container ce61e1524a8c7df6e5c170b8ee42f656f814f5b5dfe3d29c98d3533bacbd9b26: Status 404 returned error can't find the container with id ce61e1524a8c7df6e5c170b8ee42f656f814f5b5dfe3d29c98d3533bacbd9b26 Jul 11 23:48:44.763361 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:44.763332 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" event={"ID":"45e054f7-67f1-4acf-a65a-0569e16f8105","Type":"ContainerStarted","Data":"ce61e1524a8c7df6e5c170b8ee42f656f814f5b5dfe3d29c98d3533bacbd9b26"} Jul 11 23:48:45.769932 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:45.769901 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" event={"ID":"45e054f7-67f1-4acf-a65a-0569e16f8105","Type":"ContainerStarted","Data":"40d36011c2d43439b9a1744ec5ede8d46fc55ac653a7e116359e31bf1a4aad33"} Jul 11 23:48:45.770286 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:45.769966 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:48:45.795266 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:45.795214 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" podStartSLOduration=1.795198853 podStartE2EDuration="1.795198853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-11 23:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-11 23:48:45.793541394 +0000 UTC m=+1061.449164644" watchObservedRunningTime="2026-07-11 23:48:45.795198853 +0000 UTC m=+1061.450822106" Jul 11 23:48:56.776428 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:48:56.776385 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-vpv4f" Jul 11 23:50:21.884744 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:50:21.884707 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:51:04.764052 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:51:04.764015 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:51:04.768420 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:51:04.768401 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:51:04.770929 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:51:04.770909 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:51:04.774978 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:51:04.774961 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:54:34.250415 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:54:34.250382 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:54:43.740814 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:54:43.740780 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:54:51.991696 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:54:51.991662 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:54:57.053206 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:54:57.053128 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:54:59.948914 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:54:59.948868 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:00.042663 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:00.042628 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:00.389554 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:00.389449 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:06.097647 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:06.097610 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:06.154653 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:06.154620 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:06.289416 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:06.289391 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:06.343125 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:06.343084 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:07.381147 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:07.381111 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:07.470182 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:07.470150 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:07.535335 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:07.535304 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:07.878045 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:07.878005 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:08.051461 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:08.051434 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:08.152247 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:08.152219 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:55:09.855871 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:55:09.855832 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:04.793754 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:04.793691 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:56:04.798368 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:04.798346 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:56:04.804698 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:04.804679 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 11 23:56:04.810282 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:04.810261 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 11 23:56:13.548643 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:13.548603 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:17.538032 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:17.537993 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:23.155052 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:23.155016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:23.212076 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:23.212045 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:27.733236 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:27.733160 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:28.103166 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:28.103088 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:28.206094 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:28.206067 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:28.295367 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:28.295335 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:29.504212 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:29.504183 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:29.743050 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:29.743018 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:29.844289 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:29.844173 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:30.159201 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:30.159167 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:30.257192 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:30.257164 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:41.236468 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:41.236437 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:41.336646 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:41.336615 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:41.771296 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:41.771269 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:56:44.739378 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:56:44.739342 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:20.387403 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:20.387371 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:20.437564 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:20.437534 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:34.145595 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:34.145561 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:34.255615 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:34.255586 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:39.338370 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:39.338339 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:39.884109 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:39.884076 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:40.477796 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:40.477767 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:45.262140 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:45.262098 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:45.317490 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:45.317463 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:45.487985 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:45.487955 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:45.586118 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:45.586051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:46.579219 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:46.579190 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:46.670837 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:46.670810 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:46.732665 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:46.732635 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:46.951205 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:46.951173 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:47.188472 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:47.188439 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:47.289185 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:47.289115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:57:48.676572 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:57:48.676545 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:58:23.358425 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:58:23.358393 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:58:23.459114 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:58:23.459079 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:58:52.207355 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:58:52.207324 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:58:52.307544 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:58:52.307492 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:02.198514 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:02.198464 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:06.755613 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:06.755577 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:12.403551 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:12.403514 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:12.504900 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:12.504871 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:17.745522 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:17.745473 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:18.024001 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:18.023973 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:19.129182 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:19.129152 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:19.210850 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:19.210820 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:19.286486 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:19.286457 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:19.524913 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:19.524886 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:19.835232 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:19.835157 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:19.932342 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:19.932294 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:34.698189 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:34.698154 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:34.750621 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:34.750588 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:38.080881 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:38.080846 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 11 23:59:59.115443 ip-10-0-137-251 kubenswrapper[2566]: I0711 23:59:59.115389 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:15.415916 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:15.415855 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:33.923349 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:33.923311 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:38.639458 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:38.639428 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:44.259114 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:44.259071 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:44.359337 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:44.359310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:48.827677 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:48.827647 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:49.414220 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:49.414158 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:49.511096 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:49.511064 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:49.608785 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:49.608724 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:50.995400 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:50.995367 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:55.362859 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:55.362833 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:56.627910 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:56.627830 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:56.734895 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:56.734870 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:56.827936 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:56.827905 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:56.928308 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:56.928283 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:00:57.310298 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:00:57.310215 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:04.826699 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:04.826665 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:01:04.831211 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:04.831188 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:01:04.835359 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:04.835342 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:01:04.839293 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:04.839276 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:01:23.271479 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:23.271444 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:28.752714 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:28.752674 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:28.807276 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:28.807248 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:28.892177 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:28.892152 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:28.991265 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:28.991237 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:44.825027 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:44.824994 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:50.345290 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:50.345259 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:50.400704 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:50.400676 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:50.496701 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:50.496673 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:01:50.592029 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:01:50.591999 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:01.071269 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:01.071231 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:01.875739 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:01.875704 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:07.377137 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:07.377103 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:07.478313 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:07.478283 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:07.576930 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:07.576906 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:07.677054 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:07.677028 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:08.437529 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:08.437480 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:08.529287 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:08.529254 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:08.630144 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:08.630116 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:10.237043 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:10.237010 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:16.077022 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:16.076985 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:16.134396 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:16.134370 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:16.721482 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:16.721451 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:22.386197 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:22.386161 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:22.486818 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:22.486790 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:22.619228 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:22.619197 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:22.909060 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:22.909030 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:23.294470 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:23.294390 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:25.659327 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:25.659284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:25.738988 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:25.738955 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:25.808546 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:25.808519 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:25.907607 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:25.907571 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:26.195716 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:26.195686 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:26.300391 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:26.300322 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:30.170082 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:30.170050 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:30.834586 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:30.834557 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:31.096151 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:31.096072 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:31.302069 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:31.302038 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:36.908115 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:36.908082 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:36.961511 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:36.961473 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:47.626257 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:47.626219 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:48.686299 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:48.686271 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:48.742025 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:48.741995 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:52.318655 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:52.318627 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:57.955535 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:57.955483 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:02:58.011010 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:02:58.010978 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:02.984109 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:02.984075 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:03.206095 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:03.206061 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:03.317394 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:03.317317 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:03.418314 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:03.418287 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:03.554856 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:03.554816 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:09.429576 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:09.429535 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:09.497272 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:09.497241 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:09.563983 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:09.563955 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:09.664215 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:09.664188 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:09.764485 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:09.764410 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:15.248147 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:15.248115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:15.306292 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:15.306262 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:21.960755 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:21.960719 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:22.266810 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:22.266730 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:22.361344 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:22.361310 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:24.123764 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:24.123720 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:24.224303 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:24.224271 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:24.660363 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:24.660327 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:24.743951 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:24.743922 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:24.801318 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:24.801277 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:25.169879 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:25.169845 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:25.227751 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:25.227725 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:29.144366 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:29.144331 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:53.013700 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.013666 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht"] Jul 12 00:03:53.017368 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.017349 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.022268 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.021599 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-14eebdc1\"/\"default-dockercfg-srd97\"" Jul 12 00:03:53.022268 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.021702 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-14eebdc1\"/\"openshift-service-ca.crt\"" Jul 12 00:03:53.022268 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.021972 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-14eebdc1\"/\"kube-root-ca.crt\"" Jul 12 00:03:53.022268 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.022145 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-14eebdc1\"/\"e2e-lifecycle-model-14eebdc1-kserve-self-signed-certs\"" Jul 12 00:03:53.024803 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.024778 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht"] Jul 12 00:03:53.048116 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.048091 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:03:53.091332 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.091300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgtx\" (UniqueName: \"kubernetes.io/projected/2a3189ae-964a-4cda-a644-8e7a8da75d41-kube-api-access-tdgtx\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.091529 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.091344 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-tmp-dir\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.091529 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.091388 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-home\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.091529 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.091425 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-dshm\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.091529 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.091494 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3189ae-964a-4cda-a644-8e7a8da75d41-tls-certs\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.091783 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.091615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-model-cache\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192254 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192232 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-model-cache\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192372 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgtx\" (UniqueName: \"kubernetes.io/projected/2a3189ae-964a-4cda-a644-8e7a8da75d41-kube-api-access-tdgtx\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192372 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-tmp-dir\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192449 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-home\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192493 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-dshm\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192595 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192570 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3189ae-964a-4cda-a644-8e7a8da75d41-tls-certs\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192702 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192627 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-tmp-dir\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192746 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192713 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-model-cache\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.192827 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.192808 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-home\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.194681 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.194656 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-dshm\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.195057 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.195035 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3189ae-964a-4cda-a644-8e7a8da75d41-tls-certs\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.199807 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.199783 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgtx\" (UniqueName: \"kubernetes.io/projected/2a3189ae-964a-4cda-a644-8e7a8da75d41-kube-api-access-tdgtx\") pod \"e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.328542 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.328486 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:03:53.456927 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.456900 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht"] Jul 12 00:03:53.459909 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:03:53.459871 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a3189ae_964a_4cda_a644_8e7a8da75d41.slice/crio-65d09072bd032c384ca931891e45c8dd6fa63b2a17aedc8faf784e6c35f98cbc WatchSource:0}: Error finding container 65d09072bd032c384ca931891e45c8dd6fa63b2a17aedc8faf784e6c35f98cbc: Status 404 returned error can't find the container with id 65d09072bd032c384ca931891e45c8dd6fa63b2a17aedc8faf784e6c35f98cbc Jul 12 00:03:53.959432 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.959388 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" event={"ID":"2a3189ae-964a-4cda-a644-8e7a8da75d41","Type":"ContainerStarted","Data":"534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65"} Jul 12 00:03:53.959627 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.959439 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" event={"ID":"2a3189ae-964a-4cda-a644-8e7a8da75d41","Type":"ContainerStarted","Data":"65d09072bd032c384ca931891e45c8dd6fa63b2a17aedc8faf784e6c35f98cbc"} Jul 12 00:03:53.975436 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:03:53.975389 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" podStartSLOduration=1.975376475 podStartE2EDuration="1.975376475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:03:53.974265528 +0000 UTC m=+1969.629888776" watchObservedRunningTime="2026-07-12 00:03:53.975376475 +0000 UTC m=+1969.630999727" Jul 12 00:04:03.329310 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:03.329232 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:04:03.329310 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:03.329278 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:04:03.341840 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:03.341816 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:04:04.018203 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:04.018177 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:04:28.018813 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:28.018779 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:04:32.601417 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:32.601380 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:04:37.591812 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:37.591774 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht"] Jul 12 00:04:37.690280 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:37.690251 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:04:52.612497 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.612454 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" podUID="2a3189ae-964a-4cda-a644-8e7a8da75d41" containerName="main" containerID="cri-o://534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65" gracePeriod=45 Jul 12 00:04:52.859808 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.859784 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:04:52.969971 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.969949 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-model-cache\") pod \"2a3189ae-964a-4cda-a644-8e7a8da75d41\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " Jul 12 00:04:52.970113 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.969992 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-dshm\") pod \"2a3189ae-964a-4cda-a644-8e7a8da75d41\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " Jul 12 00:04:52.970113 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970015 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-home\") pod \"2a3189ae-964a-4cda-a644-8e7a8da75d41\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " Jul 12 00:04:52.970113 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970080 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3189ae-964a-4cda-a644-8e7a8da75d41-tls-certs\") pod \"2a3189ae-964a-4cda-a644-8e7a8da75d41\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " Jul 12 00:04:52.970288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970120 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdgtx\" (UniqueName: \"kubernetes.io/projected/2a3189ae-964a-4cda-a644-8e7a8da75d41-kube-api-access-tdgtx\") pod \"2a3189ae-964a-4cda-a644-8e7a8da75d41\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " Jul 12 00:04:52.970288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970170 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-tmp-dir\") pod \"2a3189ae-964a-4cda-a644-8e7a8da75d41\" (UID: \"2a3189ae-964a-4cda-a644-8e7a8da75d41\") " Jul 12 00:04:52.970288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970199 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a3189ae-964a-4cda-a644-8e7a8da75d41" (UID: "2a3189ae-964a-4cda-a644-8e7a8da75d41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:04:52.970441 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970280 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-home" (OuterVolumeSpecName: "home") pod "2a3189ae-964a-4cda-a644-8e7a8da75d41" (UID: "2a3189ae-964a-4cda-a644-8e7a8da75d41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:04:52.970496 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970448 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:04:52.970496 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970450 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a3189ae-964a-4cda-a644-8e7a8da75d41" (UID: "2a3189ae-964a-4cda-a644-8e7a8da75d41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:04:52.970496 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.970467 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:04:52.972086 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.972062 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-dshm" (OuterVolumeSpecName: "dshm") pod "2a3189ae-964a-4cda-a644-8e7a8da75d41" (UID: "2a3189ae-964a-4cda-a644-8e7a8da75d41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:04:52.972175 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.972137 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3189ae-964a-4cda-a644-8e7a8da75d41-kube-api-access-tdgtx" (OuterVolumeSpecName: "kube-api-access-tdgtx") pod "2a3189ae-964a-4cda-a644-8e7a8da75d41" (UID: "2a3189ae-964a-4cda-a644-8e7a8da75d41"). InnerVolumeSpecName "kube-api-access-tdgtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:04:52.972175 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:52.972165 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3189ae-964a-4cda-a644-8e7a8da75d41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a3189ae-964a-4cda-a644-8e7a8da75d41" (UID: "2a3189ae-964a-4cda-a644-8e7a8da75d41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:04:53.071065 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.071044 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:04:53.071065 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.071064 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a3189ae-964a-4cda-a644-8e7a8da75d41-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:04:53.071188 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.071074 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdgtx\" (UniqueName: \"kubernetes.io/projected/2a3189ae-964a-4cda-a644-8e7a8da75d41-kube-api-access-tdgtx\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:04:53.071188 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.071083 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a3189ae-964a-4cda-a644-8e7a8da75d41-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:04:53.250857 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.250793 2566 generic.go:356] "Generic (PLEG): container finished" podID="2a3189ae-964a-4cda-a644-8e7a8da75d41" containerID="534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65" exitCode=0 Jul 12 00:04:53.250857 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.250830 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" Jul 12 00:04:53.250857 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.250852 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" event={"ID":"2a3189ae-964a-4cda-a644-8e7a8da75d41","Type":"ContainerDied","Data":"534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65"} Jul 12 00:04:53.251009 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.250889 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht" event={"ID":"2a3189ae-964a-4cda-a644-8e7a8da75d41","Type":"ContainerDied","Data":"65d09072bd032c384ca931891e45c8dd6fa63b2a17aedc8faf784e6c35f98cbc"} Jul 12 00:04:53.251009 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.250908 2566 scope.go:117] "RemoveContainer" containerID="534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65" Jul 12 00:04:53.259697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.259679 2566 scope.go:117] "RemoveContainer" containerID="534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65" Jul 12 00:04:53.259934 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:04:53.259919 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65\": container with ID starting with 534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65 not found: ID does not exist" containerID="534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65" Jul 12 00:04:53.259983 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.259943 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65"} err="failed to get container status \"534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65\": rpc error: code = NotFound desc = could not find container \"534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65\": container with ID starting with 534e5072f9a7c71cd1b6f94cb4d5d812c8ea66a9d43eaf79cf7291aaf39a8a65 not found: ID does not exist" Jul 12 00:04:53.271290 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.271265 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht"] Jul 12 00:04:53.274155 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:53.274137 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-14eebdc1/e2e-lifecycle-model-14eebdc1-kserve-7b4979b58c-2c5ht"] Jul 12 00:04:54.807129 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:04:54.807092 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3189ae-964a-4cda-a644-8e7a8da75d41" path="/var/lib/kubelet/pods/2a3189ae-964a-4cda-a644-8e7a8da75d41/volumes" Jul 12 00:05:14.260289 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:14.260254 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:15.529449 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:15.529415 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:20.299929 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:20.299891 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:34.149349 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:34.149314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:34.205713 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:34.205687 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:44.811824 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:44.811783 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:45.033680 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:45.033652 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:46.071666 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:46.071636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:46.170456 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:46.170423 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:46.271221 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:46.271186 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:46.961872 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:46.961841 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:47.058454 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:47.058424 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:47.159828 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:47.159801 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:48.474794 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:48.474767 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:54.133237 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:54.133203 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:54.191347 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:54.191316 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:54.798454 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:54.798421 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:57.277440 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:57.277406 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:05:57.324574 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:05:57.324547 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:00.444435 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:00.444402 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:00.545909 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:00.545876 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:00.645086 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:00.645032 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:04.855022 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:04.854994 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:06:04.859158 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:04.859136 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:06:04.863967 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:04.863949 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:06:04.868697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:04.868678 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:06:11.136256 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:11.136224 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:11.476433 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:11.476399 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:32.480399 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:32.480356 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:38.133537 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:38.133497 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:38.187159 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:38.187129 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:38.493216 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:38.493187 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:38.800832 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:38.800750 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:38.898795 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:38.898767 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:41.699316 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:41.699261 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:41.786787 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:41.786754 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:41.894287 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:41.894257 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:42.039614 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:42.039543 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:42.345454 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:42.345385 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:42.446169 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:42.446141 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:46.195738 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:46.195698 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:46.532304 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:46.532220 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:46.827579 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:46.826636 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:46.982962 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:46.982931 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:49.782307 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:49.782265 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:49.861288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:49.861254 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:49.928606 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:49.928574 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:50.136566 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:50.136472 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:50.321236 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:50.321202 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:50.421343 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:50.421316 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:54.305448 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:54.305401 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:59.549693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.549617 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m"] Jul 12 00:06:59.550021 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.550009 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3189ae-964a-4cda-a644-8e7a8da75d41" containerName="main" Jul 12 00:06:59.550021 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.550021 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3189ae-964a-4cda-a644-8e7a8da75d41" containerName="main" Jul 12 00:06:59.550143 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.550127 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3189ae-964a-4cda-a644-8e7a8da75d41" containerName="main" Jul 12 00:06:59.553382 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.553359 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.556218 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.556192 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7007a3\"/\"kube-root-ca.crt\"" Jul 12 00:06:59.557495 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.557476 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7007a3\"/\"test-model-7007a3-kserve-self-signed-certs\"" Jul 12 00:06:59.557649 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.557565 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7007a3\"/\"default-dockercfg-dcztc\"" Jul 12 00:06:59.557649 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.557577 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7007a3\"/\"openshift-service-ca.crt\"" Jul 12 00:06:59.564016 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.563997 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m"] Jul 12 00:06:59.586823 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.586804 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:06:59.654185 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.654160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6qq\" (UniqueName: \"kubernetes.io/projected/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-kube-api-access-7w6qq\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.654272 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.654191 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tmp-dir\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.654272 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.654209 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tls-certs\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.654350 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.654292 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-model-cache\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.654350 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.654318 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-home\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.654350 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.654335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-dshm\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755259 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755229 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-model-cache\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755385 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-home\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755385 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755279 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-dshm\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755385 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755372 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6qq\" (UniqueName: \"kubernetes.io/projected/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-kube-api-access-7w6qq\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755555 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755403 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tmp-dir\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755555 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755423 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tls-certs\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755738 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755716 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-home\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755825 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755755 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-model-cache\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.755886 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.755866 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tmp-dir\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.757669 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.757650 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-dshm\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.757951 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.757932 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tls-certs\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.764068 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.764048 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6qq\" (UniqueName: \"kubernetes.io/projected/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-kube-api-access-7w6qq\") pod \"test-model-7007a3-kserve-b4b845cfc-8q64m\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.865840 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.865753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:06:59.993466 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:06:59.993444 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m"] Jul 12 00:06:59.995611 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:06:59.995574 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4abb06e_b7af_45c9_9c71_b747ad94c1b9.slice/crio-06eb1a69cd93c3b248a775335d7ab4f0c08f2cdc7c508e3bf5125cb246cdd40c WatchSource:0}: Error finding container 06eb1a69cd93c3b248a775335d7ab4f0c08f2cdc7c508e3bf5125cb246cdd40c: Status 404 returned error can't find the container with id 06eb1a69cd93c3b248a775335d7ab4f0c08f2cdc7c508e3bf5125cb246cdd40c Jul 12 00:07:00.841359 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:00.841323 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" event={"ID":"b4abb06e-b7af-45c9-9c71-b747ad94c1b9","Type":"ContainerStarted","Data":"7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b"} Jul 12 00:07:00.841789 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:00.841370 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" event={"ID":"b4abb06e-b7af-45c9-9c71-b747ad94c1b9","Type":"ContainerStarted","Data":"06eb1a69cd93c3b248a775335d7ab4f0c08f2cdc7c508e3bf5125cb246cdd40c"} Jul 12 00:07:00.860304 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:00.860259 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" podStartSLOduration=1.86024493 podStartE2EDuration="1.86024493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:07:00.858340038 +0000 UTC m=+2156.513963294" watchObservedRunningTime="2026-07-12 00:07:00.86024493 +0000 UTC m=+2156.515868182" Jul 12 00:07:03.889967 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:03.889931 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:07:03.990982 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:03.990946 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:07:09.865998 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:09.865966 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:07:09.866517 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:09.866034 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:07:09.878570 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:09.878545 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:07:09.894343 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:09.894311 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:07:11.024216 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:11.024183 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:07:11.076211 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:11.076181 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:07:45.619812 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.619775 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh"] Jul 12 00:07:45.623290 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.623273 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.625852 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.625833 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ad0409\"/\"default-dockercfg-4bzf5\"" Jul 12 00:07:45.627136 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.627113 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ad0409\"/\"openshift-service-ca.crt\"" Jul 12 00:07:45.627538 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.627520 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ad0409\"/\"kube-root-ca.crt\"" Jul 12 00:07:45.627630 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.627572 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ad0409\"/\"test-model-ad0409-kserve-self-signed-certs\"" Jul 12 00:07:45.632410 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.632383 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh"] Jul 12 00:07:45.717008 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.716980 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tls-certs\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.717148 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.717023 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-model-cache\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.717148 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.717052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-dshm\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.717148 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.717075 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfn5s\" (UniqueName: \"kubernetes.io/projected/2c7fac9e-9c11-4770-bfa0-1a514f072d20-kube-api-access-kfn5s\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.717148 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.717093 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-home\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.717148 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.717123 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tmp-dir\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.718311 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.718291 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:07:45.817685 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.817657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tls-certs\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.817795 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.817702 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-model-cache\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.817795 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.817727 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-dshm\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.817795 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.817748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfn5s\" (UniqueName: \"kubernetes.io/projected/2c7fac9e-9c11-4770-bfa0-1a514f072d20-kube-api-access-kfn5s\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.817795 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.817775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-home\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.818084 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.817807 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tmp-dir\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.818144 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.818114 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-model-cache\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.818221 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.818201 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-home\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.818262 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.818205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tmp-dir\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.819969 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.819949 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-dshm\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.820118 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.820102 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tls-certs\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.825883 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.825864 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfn5s\" (UniqueName: \"kubernetes.io/projected/2c7fac9e-9c11-4770-bfa0-1a514f072d20-kube-api-access-kfn5s\") pod \"test-model-ad0409-kserve-74fcfd59cc-chlmh\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:45.933495 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:45.933476 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:46.068006 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:46.067896 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh"] Jul 12 00:07:46.070969 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:07:46.070933 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c7fac9e_9c11_4770_bfa0_1a514f072d20.slice/crio-4705691b4b066a28146ec60b5f5cac95633ede2c104a0b63a0ba509968661d36 WatchSource:0}: Error finding container 4705691b4b066a28146ec60b5f5cac95633ede2c104a0b63a0ba509968661d36: Status 404 returned error can't find the container with id 4705691b4b066a28146ec60b5f5cac95633ede2c104a0b63a0ba509968661d36 Jul 12 00:07:47.055597 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:47.055563 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" event={"ID":"2c7fac9e-9c11-4770-bfa0-1a514f072d20","Type":"ContainerStarted","Data":"c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b"} Jul 12 00:07:47.055953 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:47.055607 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" event={"ID":"2c7fac9e-9c11-4770-bfa0-1a514f072d20","Type":"ContainerStarted","Data":"4705691b4b066a28146ec60b5f5cac95633ede2c104a0b63a0ba509968661d36"} Jul 12 00:07:47.072646 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:47.072601 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" podStartSLOduration=2.072586334 podStartE2EDuration="2.072586334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:07:47.07032302 +0000 UTC m=+2202.725946274" watchObservedRunningTime="2026-07-12 00:07:47.072586334 +0000 UTC m=+2202.728209587" Jul 12 00:07:55.934058 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:55.934030 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:55.934409 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:55.934072 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:55.946268 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:55.946245 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:07:56.111300 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:07:56.111276 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:08:08.379011 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:08.378979 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m"] Jul 12 00:08:08.403005 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:08.402977 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:09.276193 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:09.276158 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh"] Jul 12 00:08:09.299225 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:09.299197 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:09.893850 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:09.893816 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:09.950278 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:09.950247 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:19.976870 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:19.976835 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:23.399452 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.399415 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" podUID="b4abb06e-b7af-45c9-9c71-b747ad94c1b9" containerName="main" containerID="cri-o://7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b" gracePeriod=45 Jul 12 00:08:23.652212 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.652154 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:08:23.836853 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.836823 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-home\") pod \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " Jul 12 00:08:23.837021 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.836874 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-model-cache\") pod \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " Jul 12 00:08:23.837021 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.836916 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tls-certs\") pod \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " Jul 12 00:08:23.837021 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.836935 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tmp-dir\") pod \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " Jul 12 00:08:23.837021 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.836970 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w6qq\" (UniqueName: \"kubernetes.io/projected/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-kube-api-access-7w6qq\") pod \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " Jul 12 00:08:23.837021 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.836985 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-dshm\") pod \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\" (UID: \"b4abb06e-b7af-45c9-9c71-b747ad94c1b9\") " Jul 12 00:08:23.837305 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.837148 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-home" (OuterVolumeSpecName: "home") pod "b4abb06e-b7af-45c9-9c71-b747ad94c1b9" (UID: "b4abb06e-b7af-45c9-9c71-b747ad94c1b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:23.837305 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.837204 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4abb06e-b7af-45c9-9c71-b747ad94c1b9" (UID: "b4abb06e-b7af-45c9-9c71-b747ad94c1b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:23.837305 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.837219 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4abb06e-b7af-45c9-9c71-b747ad94c1b9" (UID: "b4abb06e-b7af-45c9-9c71-b747ad94c1b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:23.837305 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.837237 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:23.839127 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.839105 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-dshm" (OuterVolumeSpecName: "dshm") pod "b4abb06e-b7af-45c9-9c71-b747ad94c1b9" (UID: "b4abb06e-b7af-45c9-9c71-b747ad94c1b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:23.839259 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.839168 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4abb06e-b7af-45c9-9c71-b747ad94c1b9" (UID: "b4abb06e-b7af-45c9-9c71-b747ad94c1b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:08:23.839259 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.839172 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-kube-api-access-7w6qq" (OuterVolumeSpecName: "kube-api-access-7w6qq") pod "b4abb06e-b7af-45c9-9c71-b747ad94c1b9" (UID: "b4abb06e-b7af-45c9-9c71-b747ad94c1b9"). InnerVolumeSpecName "kube-api-access-7w6qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:08:23.938480 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.938401 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:23.938480 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.938428 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:23.938480 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.938436 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:23.938480 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.938445 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w6qq\" (UniqueName: \"kubernetes.io/projected/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-kube-api-access-7w6qq\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:23.938480 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:23.938455 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4abb06e-b7af-45c9-9c71-b747ad94c1b9-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:24.229456 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.229367 2566 generic.go:356] "Generic (PLEG): container finished" podID="b4abb06e-b7af-45c9-9c71-b747ad94c1b9" containerID="7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b" exitCode=0 Jul 12 00:08:24.229456 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.229427 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" Jul 12 00:08:24.229651 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.229455 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" event={"ID":"b4abb06e-b7af-45c9-9c71-b747ad94c1b9","Type":"ContainerDied","Data":"7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b"} Jul 12 00:08:24.229651 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.229525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m" event={"ID":"b4abb06e-b7af-45c9-9c71-b747ad94c1b9","Type":"ContainerDied","Data":"06eb1a69cd93c3b248a775335d7ab4f0c08f2cdc7c508e3bf5125cb246cdd40c"} Jul 12 00:08:24.229651 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.229546 2566 scope.go:117] "RemoveContainer" containerID="7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b" Jul 12 00:08:24.238294 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.238276 2566 scope.go:117] "RemoveContainer" containerID="7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b" Jul 12 00:08:24.238563 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:08:24.238536 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b\": container with ID starting with 7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b not found: ID does not exist" containerID="7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b" Jul 12 00:08:24.238666 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.238569 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b"} err="failed to get container status \"7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b\": rpc error: code = NotFound desc = could not find container \"7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b\": container with ID starting with 7c8fcdc358cbf4c16359622eb70251caa011a7737ae516723c6ebf92cab4b01b not found: ID does not exist" Jul 12 00:08:24.249922 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.249899 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m"] Jul 12 00:08:24.251820 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.251800 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7007a3/test-model-7007a3-kserve-b4b845cfc-8q64m"] Jul 12 00:08:24.299439 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.299414 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" podUID="2c7fac9e-9c11-4770-bfa0-1a514f072d20" containerName="main" containerID="cri-o://c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b" gracePeriod=45 Jul 12 00:08:24.526160 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.526131 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:24.546162 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.546141 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:08:24.643933 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.643907 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tmp-dir\") pod \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " Jul 12 00:08:24.644055 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.643992 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-home\") pod \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " Jul 12 00:08:24.644055 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644024 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-model-cache\") pod \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " Jul 12 00:08:24.644055 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644043 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfn5s\" (UniqueName: \"kubernetes.io/projected/2c7fac9e-9c11-4770-bfa0-1a514f072d20-kube-api-access-kfn5s\") pod \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " Jul 12 00:08:24.644225 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644066 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tls-certs\") pod \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " Jul 12 00:08:24.644225 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644109 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-dshm\") pod \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\" (UID: \"2c7fac9e-9c11-4770-bfa0-1a514f072d20\") " Jul 12 00:08:24.644225 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644138 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c7fac9e-9c11-4770-bfa0-1a514f072d20" (UID: "2c7fac9e-9c11-4770-bfa0-1a514f072d20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:24.644375 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644252 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-home" (OuterVolumeSpecName: "home") pod "2c7fac9e-9c11-4770-bfa0-1a514f072d20" (UID: "2c7fac9e-9c11-4770-bfa0-1a514f072d20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:24.644375 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644293 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c7fac9e-9c11-4770-bfa0-1a514f072d20" (UID: "2c7fac9e-9c11-4770-bfa0-1a514f072d20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:24.644375 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644356 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:24.644375 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.644369 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:24.646304 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.646285 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c7fac9e-9c11-4770-bfa0-1a514f072d20" (UID: "2c7fac9e-9c11-4770-bfa0-1a514f072d20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:08:24.646304 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.646292 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7fac9e-9c11-4770-bfa0-1a514f072d20-kube-api-access-kfn5s" (OuterVolumeSpecName: "kube-api-access-kfn5s") pod "2c7fac9e-9c11-4770-bfa0-1a514f072d20" (UID: "2c7fac9e-9c11-4770-bfa0-1a514f072d20"). InnerVolumeSpecName "kube-api-access-kfn5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:08:24.646431 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.646288 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-dshm" (OuterVolumeSpecName: "dshm") pod "2c7fac9e-9c11-4770-bfa0-1a514f072d20" (UID: "2c7fac9e-9c11-4770-bfa0-1a514f072d20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:08:24.744934 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.744912 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:24.744934 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.744934 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfn5s\" (UniqueName: \"kubernetes.io/projected/2c7fac9e-9c11-4770-bfa0-1a514f072d20-kube-api-access-kfn5s\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:24.745071 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.744945 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c7fac9e-9c11-4770-bfa0-1a514f072d20-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:24.745071 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.744953 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c7fac9e-9c11-4770-bfa0-1a514f072d20-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:08:24.808928 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.808872 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4abb06e-b7af-45c9-9c71-b747ad94c1b9" path="/var/lib/kubelet/pods/b4abb06e-b7af-45c9-9c71-b747ad94c1b9/volumes" Jul 12 00:08:24.902880 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:24.902854 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:25.000600 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.000573 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:25.235437 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.235408 2566 generic.go:356] "Generic (PLEG): container finished" podID="2c7fac9e-9c11-4770-bfa0-1a514f072d20" containerID="c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b" exitCode=0 Jul 12 00:08:25.235605 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.235448 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" Jul 12 00:08:25.235605 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.235494 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" event={"ID":"2c7fac9e-9c11-4770-bfa0-1a514f072d20","Type":"ContainerDied","Data":"c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b"} Jul 12 00:08:25.235605 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.235558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh" event={"ID":"2c7fac9e-9c11-4770-bfa0-1a514f072d20","Type":"ContainerDied","Data":"4705691b4b066a28146ec60b5f5cac95633ede2c104a0b63a0ba509968661d36"} Jul 12 00:08:25.235605 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.235582 2566 scope.go:117] "RemoveContainer" containerID="c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b" Jul 12 00:08:25.243739 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.243722 2566 scope.go:117] "RemoveContainer" containerID="c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b" Jul 12 00:08:25.243970 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:08:25.243953 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b\": container with ID starting with c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b not found: ID does not exist" containerID="c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b" Jul 12 00:08:25.244028 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.243977 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b"} err="failed to get container status \"c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b\": rpc error: code = NotFound desc = could not find container \"c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b\": container with ID starting with c989a3a816aa89641e99d28aee7e368351c16a44985921e4ebacbb58259de91b not found: ID does not exist" Jul 12 00:08:25.251745 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.251723 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh"] Jul 12 00:08:25.255327 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:25.255310 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ad0409/test-model-ad0409-kserve-74fcfd59cc-chlmh"] Jul 12 00:08:26.807811 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:26.807740 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c7fac9e-9c11-4770-bfa0-1a514f072d20" path="/var/lib/kubelet/pods/2c7fac9e-9c11-4770-bfa0-1a514f072d20/volumes" Jul 12 00:08:30.176266 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:30.176237 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:30.307923 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:30.307896 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:35.254254 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:35.254226 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:35.488680 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:35.488645 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:36.095693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:36.095655 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:36.194410 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:36.194372 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:46.000386 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:46.000342 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:49.538603 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:49.538572 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:49.586057 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:49.586014 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:50.684415 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:50.684377 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:56.417178 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:56.417147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:08:56.529944 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:08:56.529902 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:01.388545 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:01.388489 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:01.640828 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:01.640742 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:02.018693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:02.018617 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:02.335905 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:02.335829 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:02.519787 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:02.519756 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:05.509613 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:05.509560 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:05.615835 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:05.615797 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:05.709150 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:05.709121 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:05.945328 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:05.945299 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:06.171886 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:06.171848 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:06.269411 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:06.269335 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:10.068723 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:10.068687 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:10.312959 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:10.312925 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:10.558757 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:10.558722 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:10.898350 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:10.898308 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:11.057754 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:11.057719 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:13.542405 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.542371 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:13.642362 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.642335 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:13.744839 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.744807 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9"] Jul 12 00:09:13.745370 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.745355 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7fac9e-9c11-4770-bfa0-1a514f072d20" containerName="main" Jul 12 00:09:13.745419 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.745376 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7fac9e-9c11-4770-bfa0-1a514f072d20" containerName="main" Jul 12 00:09:13.745419 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.745399 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4abb06e-b7af-45c9-9c71-b747ad94c1b9" containerName="main" Jul 12 00:09:13.745419 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.745409 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4abb06e-b7af-45c9-9c71-b747ad94c1b9" containerName="main" Jul 12 00:09:13.745536 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.745528 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4abb06e-b7af-45c9-9c71-b747ad94c1b9" containerName="main" Jul 12 00:09:13.745572 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.745543 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c7fac9e-9c11-4770-bfa0-1a514f072d20" containerName="main" Jul 12 00:09:13.748964 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.748948 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.751660 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.751634 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a58284-serving-cert\"" Jul 12 00:09:13.751839 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.751817 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6n9rc\"" Jul 12 00:09:13.758442 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.758420 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9"] Jul 12 00:09:13.778853 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.778826 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:13.853940 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.853874 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.853940 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.853926 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6cf\" (UniqueName: \"kubernetes.io/projected/de919509-7a43-4316-91f0-df922f4cb01f-kube-api-access-qc6cf\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.854084 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.853975 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de919509-7a43-4316-91f0-df922f4cb01f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.955520 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.955470 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.955679 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.955537 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6cf\" (UniqueName: \"kubernetes.io/projected/de919509-7a43-4316-91f0-df922f4cb01f-kube-api-access-qc6cf\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.955679 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.955561 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de919509-7a43-4316-91f0-df922f4cb01f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.955679 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:09:13.955638 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-serving-cert: secret "maas-api-e2e-api-b-a58284-serving-cert" not found Jul 12 00:09:13.955808 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:09:13.955742 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls podName:de919509-7a43-4316-91f0-df922f4cb01f nodeName:}" failed. No retries permitted until 2026-07-12 00:09:14.45572208 +0000 UTC m=+2290.111345324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls") pod "maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" (UID: "de919509-7a43-4316-91f0-df922f4cb01f") : secret "maas-api-e2e-api-b-a58284-serving-cert" not found Jul 12 00:09:13.956149 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.956133 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de919509-7a43-4316-91f0-df922f4cb01f-openshift-service-ca\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:13.964011 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:13.963984 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6cf\" (UniqueName: \"kubernetes.io/projected/de919509-7a43-4316-91f0-df922f4cb01f-kube-api-access-qc6cf\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:14.191215 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.191185 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:14.294492 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.294458 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:14.391463 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.391438 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:14.459296 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.459226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:14.461607 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.461591 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls\") pod \"maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:14.662362 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.662336 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:14.796476 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.796452 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9"] Jul 12 00:09:14.798276 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:09:14.798248 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde919509_7a43_4316_91f0_df922f4cb01f.slice/crio-1a244e2c7fcfb7c851b38e80221f76034bc519e92b97e34099dbb7c4bad56715 WatchSource:0}: Error finding container 1a244e2c7fcfb7c851b38e80221f76034bc519e92b97e34099dbb7c4bad56715: Status 404 returned error can't find the container with id 1a244e2c7fcfb7c851b38e80221f76034bc519e92b97e34099dbb7c4bad56715 Jul 12 00:09:14.799716 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:14.799700 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 00:09:15.488817 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:15.488780 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" event={"ID":"de919509-7a43-4316-91f0-df922f4cb01f","Type":"ContainerStarted","Data":"1a244e2c7fcfb7c851b38e80221f76034bc519e92b97e34099dbb7c4bad56715"} Jul 12 00:09:17.501096 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:17.501064 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" event={"ID":"de919509-7a43-4316-91f0-df922f4cb01f","Type":"ContainerStarted","Data":"ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902"} Jul 12 00:09:17.501532 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:17.501187 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:17.517184 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:17.517126 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" podStartSLOduration=2.37914011 podStartE2EDuration="4.517100723s" totalImagesPullingTime="2.137960613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:09:17.516128925 +0000 UTC m=+2293.171752191" watchObservedRunningTime="2026-07-12 00:09:17.517100723 +0000 UTC m=+2293.172723979" Jul 12 00:09:18.078262 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:18.078225 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:23.511272 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:23.511242 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:09:26.920817 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:26.920782 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:27.040361 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:27.040333 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:35.097447 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:35.097414 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:35.239537 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:35.239480 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:36.808350 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:36.808316 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:36.907879 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:36.907848 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:37.045416 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:37.045383 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:41.400208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:41.400178 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:47.000332 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:47.000295 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:47.065617 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:47.065583 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:52.099458 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:52.099427 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:52.338745 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:52.338713 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:53.015292 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:53.015258 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:09:53.116073 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:09:53.116040 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:02.832855 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:02.832763 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9"] Jul 12 00:10:02.833760 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:02.833000 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" podUID="de919509-7a43-4316-91f0-df922f4cb01f" containerName="maas-api" containerID="cri-o://ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902" gracePeriod=30 Jul 12 00:10:02.854094 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:02.854072 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:03.072078 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.072058 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:10:03.248498 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.248463 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de919509-7a43-4316-91f0-df922f4cb01f-openshift-service-ca\") pod \"de919509-7a43-4316-91f0-df922f4cb01f\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " Jul 12 00:10:03.248684 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.248528 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls\") pod \"de919509-7a43-4316-91f0-df922f4cb01f\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " Jul 12 00:10:03.248684 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.248604 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc6cf\" (UniqueName: \"kubernetes.io/projected/de919509-7a43-4316-91f0-df922f4cb01f-kube-api-access-qc6cf\") pod \"de919509-7a43-4316-91f0-df922f4cb01f\" (UID: \"de919509-7a43-4316-91f0-df922f4cb01f\") " Jul 12 00:10:03.248866 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.248836 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de919509-7a43-4316-91f0-df922f4cb01f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de919509-7a43-4316-91f0-df922f4cb01f" (UID: "de919509-7a43-4316-91f0-df922f4cb01f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 00:10:03.250664 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.250644 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de919509-7a43-4316-91f0-df922f4cb01f" (UID: "de919509-7a43-4316-91f0-df922f4cb01f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:10:03.250750 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.250698 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de919509-7a43-4316-91f0-df922f4cb01f-kube-api-access-qc6cf" (OuterVolumeSpecName: "kube-api-access-qc6cf") pod "de919509-7a43-4316-91f0-df922f4cb01f" (UID: "de919509-7a43-4316-91f0-df922f4cb01f"). InnerVolumeSpecName "kube-api-access-qc6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:10:03.349172 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.349149 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6cf\" (UniqueName: \"kubernetes.io/projected/de919509-7a43-4316-91f0-df922f4cb01f-kube-api-access-qc6cf\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:10:03.349172 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.349171 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de919509-7a43-4316-91f0-df922f4cb01f-openshift-service-ca\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:10:03.349299 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.349181 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de919509-7a43-4316-91f0-df922f4cb01f-maas-api-tls\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:10:03.724371 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.724339 2566 generic.go:356] "Generic (PLEG): container finished" podID="de919509-7a43-4316-91f0-df922f4cb01f" containerID="ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902" exitCode=0 Jul 12 00:10:03.724523 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.724377 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" Jul 12 00:10:03.724523 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.724426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" event={"ID":"de919509-7a43-4316-91f0-df922f4cb01f","Type":"ContainerDied","Data":"ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902"} Jul 12 00:10:03.724523 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.724467 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9" event={"ID":"de919509-7a43-4316-91f0-df922f4cb01f","Type":"ContainerDied","Data":"1a244e2c7fcfb7c851b38e80221f76034bc519e92b97e34099dbb7c4bad56715"} Jul 12 00:10:03.724523 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.724488 2566 scope.go:117] "RemoveContainer" containerID="ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902" Jul 12 00:10:03.733598 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.733581 2566 scope.go:117] "RemoveContainer" containerID="ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902" Jul 12 00:10:03.733834 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:10:03.733816 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902\": container with ID starting with ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902 not found: ID does not exist" containerID="ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902" Jul 12 00:10:03.733884 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.733842 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902"} err="failed to get container status \"ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902\": rpc error: code = NotFound desc = could not find container \"ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902\": container with ID starting with ed76fb77997ce7d1932cae7161da13e8e68d9f97fd5d9d0d3b387c48be5dc902 not found: ID does not exist" Jul 12 00:10:03.745337 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.745316 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9"] Jul 12 00:10:03.748550 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:03.748530 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a58284-fbf76bd8c-7k9q9"] Jul 12 00:10:04.807149 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:04.807120 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de919509-7a43-4316-91f0-df922f4cb01f" path="/var/lib/kubelet/pods/de919509-7a43-4316-91f0-df922f4cb01f/volumes" Jul 12 00:10:07.537173 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:07.537137 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:13.073588 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:13.073530 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:13.126450 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:13.126428 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:18.289744 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:18.289707 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:19.755049 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:19.755016 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:21.898945 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.898909 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:21.971404 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.971377 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j"] Jul 12 00:10:21.971813 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.971798 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de919509-7a43-4316-91f0-df922f4cb01f" containerName="maas-api" Jul 12 00:10:21.971867 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.971815 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="de919509-7a43-4316-91f0-df922f4cb01f" containerName="maas-api" Jul 12 00:10:21.971915 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.971888 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="de919509-7a43-4316-91f0-df922f4cb01f" containerName="maas-api" Jul 12 00:10:21.975032 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.975010 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:21.977640 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.977620 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a7d1ef-serving-cert\"" Jul 12 00:10:21.977746 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.977682 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-6n9rc\"" Jul 12 00:10:21.980830 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.980796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:21.995248 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.995220 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j"] Jul 12 00:10:21.999521 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.999482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f9075-2c6f-4391-ab69-3a12d3658e8b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:21.999636 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.999614 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:21.999683 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:21.999659 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzpwq\" (UniqueName: \"kubernetes.io/projected/b18f9075-2c6f-4391-ab69-3a12d3658e8b-kube-api-access-mzpwq\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:22.092971 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.092940 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:22.100136 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.100105 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f9075-2c6f-4391-ab69-3a12d3658e8b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:22.100335 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.100174 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:22.100335 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.100204 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzpwq\" (UniqueName: \"kubernetes.io/projected/b18f9075-2c6f-4391-ab69-3a12d3658e8b-kube-api-access-mzpwq\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:22.100451 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:10:22.100335 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-serving-cert: secret "maas-api-e2e-shared-a-a7d1ef-serving-cert" not found Jul 12 00:10:22.100451 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:10:22.100417 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls podName:b18f9075-2c6f-4391-ab69-3a12d3658e8b nodeName:}" failed. No retries permitted until 2026-07-12 00:10:22.600395895 +0000 UTC m=+2358.256019139 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls") pod "maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" (UID: "b18f9075-2c6f-4391-ab69-3a12d3658e8b") : secret "maas-api-e2e-shared-a-a7d1ef-serving-cert" not found Jul 12 00:10:22.100828 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.100809 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f9075-2c6f-4391-ab69-3a12d3658e8b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:22.115228 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.115196 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzpwq\" (UniqueName: \"kubernetes.io/projected/b18f9075-2c6f-4391-ab69-3a12d3658e8b-kube-api-access-mzpwq\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:22.537237 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.537206 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:22.604333 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.604304 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:22.604481 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:10:22.604410 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-serving-cert: secret "maas-api-e2e-shared-a-a7d1ef-serving-cert" not found Jul 12 00:10:22.604481 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:10:22.604472 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls podName:b18f9075-2c6f-4391-ab69-3a12d3658e8b nodeName:}" failed. No retries permitted until 2026-07-12 00:10:23.604458977 +0000 UTC m=+2359.260082208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls") pod "maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" (UID: "b18f9075-2c6f-4391-ab69-3a12d3658e8b") : secret "maas-api-e2e-shared-a-a7d1ef-serving-cert" not found Jul 12 00:10:22.637994 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.637963 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:22.880689 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:22.880572 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:23.614099 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:23.614070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:23.616414 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:23.616391 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls\") pod \"maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:23.794059 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:23.794031 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:23.921909 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:23.921828 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j"] Jul 12 00:10:23.924259 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:10:23.924233 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18f9075_2c6f_4391_ab69_3a12d3658e8b.slice/crio-542f1c6b4b4b871a9f9efcf05554ad917e3e1b63a96e3f23f35da89a2cf5e76c WatchSource:0}: Error finding container 542f1c6b4b4b871a9f9efcf05554ad917e3e1b63a96e3f23f35da89a2cf5e76c: Status 404 returned error can't find the container with id 542f1c6b4b4b871a9f9efcf05554ad917e3e1b63a96e3f23f35da89a2cf5e76c Jul 12 00:10:24.827543 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:24.827447 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" event={"ID":"b18f9075-2c6f-4391-ab69-3a12d3658e8b","Type":"ContainerStarted","Data":"1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5"} Jul 12 00:10:24.827543 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:24.827493 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" event={"ID":"b18f9075-2c6f-4391-ab69-3a12d3658e8b","Type":"ContainerStarted","Data":"542f1c6b4b4b871a9f9efcf05554ad917e3e1b63a96e3f23f35da89a2cf5e76c"} Jul 12 00:10:24.827901 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:24.827550 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:24.846133 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:24.846090 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" podStartSLOduration=3.227632812 podStartE2EDuration="3.846079349s" totalImagesPullingTime="618.446537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:10:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:10:24.844389129 +0000 UTC m=+2360.500012403" watchObservedRunningTime="2026-07-12 00:10:24.846079349 +0000 UTC m=+2360.501702602" Jul 12 00:10:26.422553 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:26.422518 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:26.765146 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:26.765061 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:27.076874 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:27.076793 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:27.320219 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:27.320186 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:29.956642 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:29.956612 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:30.067643 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.067604 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:30.068755 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.068730 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp"] Jul 12 00:10:30.075588 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.075566 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.079491 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.079463 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-a2ea47-serving-cert\"" Jul 12 00:10:30.091422 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.091399 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp"] Jul 12 00:10:30.168462 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.168430 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.168462 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.168461 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64kb9\" (UniqueName: \"kubernetes.io/projected/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-kube-api-access-64kb9\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.168665 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.168526 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.239992 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.239933 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:30.269157 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.269135 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.269232 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.269168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64kb9\" (UniqueName: \"kubernetes.io/projected/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-kube-api-access-64kb9\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.269232 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.269218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.269311 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:10:30.269286 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-serving-cert: secret "maas-api-e2e-shared-b-a2ea47-serving-cert" not found Jul 12 00:10:30.269364 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:10:30.269352 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls podName:c3b7bfec-d6d8-4b41-8059-d05efe4cd435 nodeName:}" failed. No retries permitted until 2026-07-12 00:10:30.769332957 +0000 UTC m=+2366.424956187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls") pod "maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" (UID: "c3b7bfec-d6d8-4b41-8059-d05efe4cd435") : secret "maas-api-e2e-shared-b-a2ea47-serving-cert" not found Jul 12 00:10:30.269867 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.269849 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-openshift-service-ca\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.280675 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.280655 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64kb9\" (UniqueName: \"kubernetes.io/projected/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-kube-api-access-64kb9\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.499370 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.499288 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:30.634769 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.634743 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:30.734907 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.734855 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:30.773972 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.773896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.776179 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.776155 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls\") pod \"maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:30.837050 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.837020 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:10:30.987525 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:30.987476 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:31.119155 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:31.119131 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp"] Jul 12 00:10:31.121547 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:10:31.121486 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b7bfec_d6d8_4b41_8059_d05efe4cd435.slice/crio-146725b7272aea3512f9a5011e3453629b881a3353b6c00f33894797dc3618b4 WatchSource:0}: Error finding container 146725b7272aea3512f9a5011e3453629b881a3353b6c00f33894797dc3618b4: Status 404 returned error can't find the container with id 146725b7272aea3512f9a5011e3453629b881a3353b6c00f33894797dc3618b4 Jul 12 00:10:31.861839 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:31.861753 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" event={"ID":"c3b7bfec-d6d8-4b41-8059-d05efe4cd435","Type":"ContainerStarted","Data":"0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4"} Jul 12 00:10:31.861839 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:31.861809 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" event={"ID":"c3b7bfec-d6d8-4b41-8059-d05efe4cd435","Type":"ContainerStarted","Data":"146725b7272aea3512f9a5011e3453629b881a3353b6c00f33894797dc3618b4"} Jul 12 00:10:31.861839 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:31.861840 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:31.879077 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:31.879017 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" podStartSLOduration=1.545247014 podStartE2EDuration="1.879004107s" totalImagesPullingTime="333.757093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:10:31.876930642 +0000 UTC m=+2367.532553922" watchObservedRunningTime="2026-07-12 00:10:31.879004107 +0000 UTC m=+2367.534627359" Jul 12 00:10:34.485402 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:34.485363 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:37.873831 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:37.873802 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:10:43.236981 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:43.236945 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:43.337522 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:43.337470 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:50.015198 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.015167 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg"] Jul 12 00:10:50.018744 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.018726 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.023047 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.023002 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"kube-root-ca.crt\"" Jul 12 00:10:50.023047 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.023030 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"openshift-service-ca.crt\"" Jul 12 00:10:50.023359 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.023034 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"default-dockercfg-7tpxz\"" Jul 12 00:10:50.023359 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.023081 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"auth-test-model-a7d1ef-kserve-self-signed-certs\"" Jul 12 00:10:50.026871 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.026849 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg"] Jul 12 00:10:50.038637 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.038611 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:50.128560 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.128528 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x7c6\" (UniqueName: \"kubernetes.io/projected/e7a80284-e190-4a21-a8e4-33b299b4b2b9-kube-api-access-9x7c6\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.128697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.128589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-home\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.128697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.128608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tmp-dir\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.128697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.128636 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tls-certs\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.128697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.128662 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-dshm\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.128697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.128680 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-model-cache\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230055 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tmp-dir\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230170 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tls-certs\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230170 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230093 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-dshm\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230170 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230110 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-model-cache\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230170 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7c6\" (UniqueName: \"kubernetes.io/projected/e7a80284-e190-4a21-a8e4-33b299b4b2b9-kube-api-access-9x7c6\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230336 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-home\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230527 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230474 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tmp-dir\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230634 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230584 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-model-cache\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.230634 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.230590 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-home\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.232273 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.232250 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-dshm\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.232534 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.232493 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tls-certs\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.237178 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.237156 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7c6\" (UniqueName: \"kubernetes.io/projected/e7a80284-e190-4a21-a8e4-33b299b4b2b9-kube-api-access-9x7c6\") pod \"auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.330296 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.330246 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:10:50.459877 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.459854 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg"] Jul 12 00:10:50.461867 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:10:50.461839 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a80284_e190_4a21_a8e4_33b299b4b2b9.slice/crio-e0d2c84ff64cf87ab8ef3f1b78c61e56c8e80e45547eecebe22b208697b5fe15 WatchSource:0}: Error finding container e0d2c84ff64cf87ab8ef3f1b78c61e56c8e80e45547eecebe22b208697b5fe15: Status 404 returned error can't find the container with id e0d2c84ff64cf87ab8ef3f1b78c61e56c8e80e45547eecebe22b208697b5fe15 Jul 12 00:10:50.958956 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.958928 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" event={"ID":"e7a80284-e190-4a21-a8e4-33b299b4b2b9","Type":"ContainerStarted","Data":"ad91e2be3a16f8e4e0d38ee736d9579413601db96356c1cdd81e7870fa317b97"} Jul 12 00:10:50.959111 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.958963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" event={"ID":"e7a80284-e190-4a21-a8e4-33b299b4b2b9","Type":"ContainerStarted","Data":"e0d2c84ff64cf87ab8ef3f1b78c61e56c8e80e45547eecebe22b208697b5fe15"} Jul 12 00:10:50.976429 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:50.976382 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" podStartSLOduration=1.97636829 podStartE2EDuration="1.97636829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:10:50.974737228 +0000 UTC m=+2386.630360480" watchObservedRunningTime="2026-07-12 00:10:50.97636829 +0000 UTC m=+2386.631991539" Jul 12 00:10:51.387867 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:51.387792 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:10:51.489832 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:10:51.489807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:11:00.330619 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:00.330571 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:11:00.331017 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:00.330628 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:11:00.342858 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:00.342823 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:11:01.019480 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:01.019453 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:11:04.883377 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:04.883352 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:11:04.887622 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:04.887597 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:11:04.894658 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:04.894638 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:11:04.898401 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:04.898384 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:11:17.821262 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.821224 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6"] Jul 12 00:11:17.825200 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.825181 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.827741 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.827715 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"auth-test-model-a2ea47-kserve-self-signed-certs\"" Jul 12 00:11:17.830130 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.830099 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"default-dockercfg-5w5pk\"" Jul 12 00:11:17.833531 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.830309 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"openshift-service-ca.crt\"" Jul 12 00:11:17.833531 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.830906 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"kube-root-ca.crt\"" Jul 12 00:11:17.837540 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.837476 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6"] Jul 12 00:11:17.851321 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.851300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-model-cache\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.851418 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.851331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-home\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.851418 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.851350 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tmp-dir\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.851527 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.851472 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-dshm\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.851578 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.851532 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdckl\" (UniqueName: \"kubernetes.io/projected/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-kube-api-access-gdckl\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.851631 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.851583 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tls-certs\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.868137 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.868115 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:11:17.952281 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952251 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdckl\" (UniqueName: \"kubernetes.io/projected/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-kube-api-access-gdckl\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952382 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952295 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tls-certs\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952382 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952329 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-model-cache\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952382 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-home\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952382 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tmp-dir\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952614 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952411 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-dshm\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952813 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952792 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-model-cache\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952813 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952805 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-home\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.952906 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.952848 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tmp-dir\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.954603 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.954587 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-dshm\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.954859 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.954843 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tls-certs\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:17.960234 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:17.960214 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdckl\" (UniqueName: \"kubernetes.io/projected/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-kube-api-access-gdckl\") pod \"auth-test-model-a2ea47-kserve-59dd49447f-mcxp6\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:18.143645 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:18.143621 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:18.277998 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:18.277969 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6"] Jul 12 00:11:18.280744 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:11:18.280718 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5b0147b_380f_4cf4_9e81_46c567b3fc0c.slice/crio-5b57d923f7543f960178f4033f270e897084dafbed5f348eab4a82e2c45dccf3 WatchSource:0}: Error finding container 5b57d923f7543f960178f4033f270e897084dafbed5f348eab4a82e2c45dccf3: Status 404 returned error can't find the container with id 5b57d923f7543f960178f4033f270e897084dafbed5f348eab4a82e2c45dccf3 Jul 12 00:11:19.099048 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:19.099013 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" event={"ID":"b5b0147b-380f-4cf4-9e81-46c567b3fc0c","Type":"ContainerStarted","Data":"eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb"} Jul 12 00:11:19.099492 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:19.099063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" event={"ID":"b5b0147b-380f-4cf4-9e81-46c567b3fc0c","Type":"ContainerStarted","Data":"5b57d923f7543f960178f4033f270e897084dafbed5f348eab4a82e2c45dccf3"} Jul 12 00:11:19.116167 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:19.116123 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" podStartSLOduration=2.116109722 podStartE2EDuration="2.116109722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:11:19.115185505 +0000 UTC m=+2414.770808759" watchObservedRunningTime="2026-07-12 00:11:19.116109722 +0000 UTC m=+2414.771732975" Jul 12 00:11:28.144377 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:28.144301 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:28.144377 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:28.144349 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:28.157153 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:28.157133 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:11:29.156330 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:11:29.156302 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:12:08.712300 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:12:08.712259 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:13:13.546650 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:13.546552 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg"] Jul 12 00:13:13.565673 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:13.565644 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:13:14.436117 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:14.436082 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6"] Jul 12 00:13:14.454704 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:14.454675 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:13:16.790543 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:16.790488 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:13:19.212870 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:19.212839 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:13:25.721850 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:25.721813 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:13:25.823178 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:25.823147 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:13:28.568476 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.568405 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" podUID="e7a80284-e190-4a21-a8e4-33b299b4b2b9" containerName="main" containerID="cri-o://ad91e2be3a16f8e4e0d38ee736d9579413601db96356c1cdd81e7870fa317b97" gracePeriod=45 Jul 12 00:13:28.717370 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.717341 2566 generic.go:356] "Generic (PLEG): container finished" podID="e7a80284-e190-4a21-a8e4-33b299b4b2b9" containerID="ad91e2be3a16f8e4e0d38ee736d9579413601db96356c1cdd81e7870fa317b97" exitCode=0 Jul 12 00:13:28.717529 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.717391 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" event={"ID":"e7a80284-e190-4a21-a8e4-33b299b4b2b9","Type":"ContainerDied","Data":"ad91e2be3a16f8e4e0d38ee736d9579413601db96356c1cdd81e7870fa317b97"} Jul 12 00:13:28.828614 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.828552 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:13:28.949468 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949442 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-home\") pod \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " Jul 12 00:13:28.949601 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949485 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tmp-dir\") pod \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " Jul 12 00:13:28.949601 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949557 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-dshm\") pod \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " Jul 12 00:13:28.949601 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949595 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x7c6\" (UniqueName: \"kubernetes.io/projected/e7a80284-e190-4a21-a8e4-33b299b4b2b9-kube-api-access-9x7c6\") pod \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " Jul 12 00:13:28.949780 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949684 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-model-cache\") pod \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " Jul 12 00:13:28.949780 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949719 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tls-certs\") pod \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\" (UID: \"e7a80284-e190-4a21-a8e4-33b299b4b2b9\") " Jul 12 00:13:28.949780 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949738 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-home" (OuterVolumeSpecName: "home") pod "e7a80284-e190-4a21-a8e4-33b299b4b2b9" (UID: "e7a80284-e190-4a21-a8e4-33b299b4b2b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:28.949780 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949772 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7a80284-e190-4a21-a8e4-33b299b4b2b9" (UID: "e7a80284-e190-4a21-a8e4-33b299b4b2b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:28.949969 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949948 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7a80284-e190-4a21-a8e4-33b299b4b2b9" (UID: "e7a80284-e190-4a21-a8e4-33b299b4b2b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:28.950003 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949964 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:28.950003 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.949988 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:28.951845 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.951811 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a80284-e190-4a21-a8e4-33b299b4b2b9-kube-api-access-9x7c6" (OuterVolumeSpecName: "kube-api-access-9x7c6") pod "e7a80284-e190-4a21-a8e4-33b299b4b2b9" (UID: "e7a80284-e190-4a21-a8e4-33b299b4b2b9"). InnerVolumeSpecName "kube-api-access-9x7c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:13:28.951845 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.951837 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-dshm" (OuterVolumeSpecName: "dshm") pod "e7a80284-e190-4a21-a8e4-33b299b4b2b9" (UID: "e7a80284-e190-4a21-a8e4-33b299b4b2b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:28.951988 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:28.951847 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7a80284-e190-4a21-a8e4-33b299b4b2b9" (UID: "e7a80284-e190-4a21-a8e4-33b299b4b2b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:13:29.050663 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.050638 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.050663 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.050660 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7a80284-e190-4a21-a8e4-33b299b4b2b9-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.050797 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.050668 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7a80284-e190-4a21-a8e4-33b299b4b2b9-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.050797 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.050676 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x7c6\" (UniqueName: \"kubernetes.io/projected/e7a80284-e190-4a21-a8e4-33b299b4b2b9-kube-api-access-9x7c6\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.456096 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.456038 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" podUID="b5b0147b-380f-4cf4-9e81-46c567b3fc0c" containerName="main" containerID="cri-o://eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb" gracePeriod=45 Jul 12 00:13:29.711785 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.711727 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:13:29.722540 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.722520 2566 generic.go:356] "Generic (PLEG): container finished" podID="b5b0147b-380f-4cf4-9e81-46c567b3fc0c" containerID="eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb" exitCode=0 Jul 12 00:13:29.722648 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.722606 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" event={"ID":"b5b0147b-380f-4cf4-9e81-46c567b3fc0c","Type":"ContainerDied","Data":"eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb"} Jul 12 00:13:29.722693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.722648 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" event={"ID":"b5b0147b-380f-4cf4-9e81-46c567b3fc0c","Type":"ContainerDied","Data":"5b57d923f7543f960178f4033f270e897084dafbed5f348eab4a82e2c45dccf3"} Jul 12 00:13:29.722693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.722666 2566 scope.go:117] "RemoveContainer" containerID="eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb" Jul 12 00:13:29.722693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.722613 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6" Jul 12 00:13:29.723803 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.723783 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" event={"ID":"e7a80284-e190-4a21-a8e4-33b299b4b2b9","Type":"ContainerDied","Data":"e0d2c84ff64cf87ab8ef3f1b78c61e56c8e80e45547eecebe22b208697b5fe15"} Jul 12 00:13:29.723912 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.723790 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg" Jul 12 00:13:29.731687 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.731671 2566 scope.go:117] "RemoveContainer" containerID="eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb" Jul 12 00:13:29.731954 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:13:29.731920 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb\": container with ID starting with eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb not found: ID does not exist" containerID="eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb" Jul 12 00:13:29.732036 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.731959 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb"} err="failed to get container status \"eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb\": rpc error: code = NotFound desc = could not find container \"eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb\": container with ID starting with eee6a04bb9c7cc41a7673974f290d12d5db86f1b62d6d0d63348ce419bffc4cb not found: ID does not exist" Jul 12 00:13:29.732036 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.731975 2566 scope.go:117] "RemoveContainer" containerID="ad91e2be3a16f8e4e0d38ee736d9579413601db96356c1cdd81e7870fa317b97" Jul 12 00:13:29.747010 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.746988 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg"] Jul 12 00:13:29.752880 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.752858 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/auth-test-model-a7d1ef-kserve-7c455b68f9-bv6rg"] Jul 12 00:13:29.756382 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756361 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-model-cache\") pod \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " Jul 12 00:13:29.756450 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756396 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-dshm\") pod \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " Jul 12 00:13:29.756450 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756435 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tmp-dir\") pod \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " Jul 12 00:13:29.756543 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756494 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tls-certs\") pod \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " Jul 12 00:13:29.756600 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756550 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-home\") pod \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " Jul 12 00:13:29.756655 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756605 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdckl\" (UniqueName: \"kubernetes.io/projected/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-kube-api-access-gdckl\") pod \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\" (UID: \"b5b0147b-380f-4cf4-9e81-46c567b3fc0c\") " Jul 12 00:13:29.756711 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756660 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5b0147b-380f-4cf4-9e81-46c567b3fc0c" (UID: "b5b0147b-380f-4cf4-9e81-46c567b3fc0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:29.756793 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756768 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5b0147b-380f-4cf4-9e81-46c567b3fc0c" (UID: "b5b0147b-380f-4cf4-9e81-46c567b3fc0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:29.756854 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756825 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-home" (OuterVolumeSpecName: "home") pod "b5b0147b-380f-4cf4-9e81-46c567b3fc0c" (UID: "b5b0147b-380f-4cf4-9e81-46c567b3fc0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:29.756959 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756943 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.757023 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756966 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.757023 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.756980 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.758612 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.758588 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-dshm" (OuterVolumeSpecName: "dshm") pod "b5b0147b-380f-4cf4-9e81-46c567b3fc0c" (UID: "b5b0147b-380f-4cf4-9e81-46c567b3fc0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:13:29.758709 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.758628 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5b0147b-380f-4cf4-9e81-46c567b3fc0c" (UID: "b5b0147b-380f-4cf4-9e81-46c567b3fc0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:13:29.758774 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.758735 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-kube-api-access-gdckl" (OuterVolumeSpecName: "kube-api-access-gdckl") pod "b5b0147b-380f-4cf4-9e81-46c567b3fc0c" (UID: "b5b0147b-380f-4cf4-9e81-46c567b3fc0c"). InnerVolumeSpecName "kube-api-access-gdckl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:13:29.858387 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.858360 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.858387 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.858385 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdckl\" (UniqueName: \"kubernetes.io/projected/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-kube-api-access-gdckl\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:29.858550 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:29.858396 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5b0147b-380f-4cf4-9e81-46c567b3fc0c-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:13:30.045029 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:30.045001 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6"] Jul 12 00:13:30.048341 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:30.048317 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/auth-test-model-a2ea47-kserve-59dd49447f-mcxp6"] Jul 12 00:13:30.806633 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:30.806605 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b0147b-380f-4cf4-9e81-46c567b3fc0c" path="/var/lib/kubelet/pods/b5b0147b-380f-4cf4-9e81-46c567b3fc0c/volumes" Jul 12 00:13:30.806990 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:13:30.806973 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a80284-e190-4a21-a8e4-33b299b4b2b9" path="/var/lib/kubelet/pods/e7a80284-e190-4a21-a8e4-33b299b4b2b9/volumes" Jul 12 00:14:15.758354 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:15.758314 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:14:22.924667 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.924637 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5"] Jul 12 00:14:22.925142 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.925123 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a80284-e190-4a21-a8e4-33b299b4b2b9" containerName="main" Jul 12 00:14:22.925142 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.925140 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a80284-e190-4a21-a8e4-33b299b4b2b9" containerName="main" Jul 12 00:14:22.925221 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.925161 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b0147b-380f-4cf4-9e81-46c567b3fc0c" containerName="main" Jul 12 00:14:22.925221 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.925172 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b0147b-380f-4cf4-9e81-46c567b3fc0c" containerName="main" Jul 12 00:14:22.925288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.925244 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a80284-e190-4a21-a8e4-33b299b4b2b9" containerName="main" Jul 12 00:14:22.925288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.925253 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b0147b-380f-4cf4-9e81-46c567b3fc0c" containerName="main" Jul 12 00:14:22.928235 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.928219 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:22.931131 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.931112 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"kube-root-ca.crt\"" Jul 12 00:14:22.932548 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.932446 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"default-dockercfg-5w5pk\"" Jul 12 00:14:22.932548 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.932488 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"openshift-service-ca.crt\"" Jul 12 00:14:22.932713 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.932566 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"sub-test-model-a2ea47-kserve-self-signed-certs\"" Jul 12 00:14:22.936293 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.936278 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5"] Jul 12 00:14:22.947281 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.947259 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:14:22.972413 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.972391 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclkb\" (UniqueName: \"kubernetes.io/projected/69a26d05-3647-43f4-a256-01f74816127a-kube-api-access-gclkb\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:22.972531 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.972446 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-dshm\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:22.972531 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.972463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-tmp-dir\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:22.972627 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.972550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-home\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:22.972627 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.972595 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a26d05-3647-43f4-a256-01f74816127a-tls-certs\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:22.972704 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:22.972645 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-model-cache\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073174 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073151 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-dshm\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073289 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-tmp-dir\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073289 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073205 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-home\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073289 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073234 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a26d05-3647-43f4-a256-01f74816127a-tls-certs\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073289 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-model-cache\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073289 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073283 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gclkb\" (UniqueName: \"kubernetes.io/projected/69a26d05-3647-43f4-a256-01f74816127a-kube-api-access-gclkb\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073697 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073668 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-home\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073805 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073744 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-tmp-dir\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.073805 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.073764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-model-cache\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.075424 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.075396 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-dshm\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.075721 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.075705 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a26d05-3647-43f4-a256-01f74816127a-tls-certs\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.081445 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.081423 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclkb\" (UniqueName: \"kubernetes.io/projected/69a26d05-3647-43f4-a256-01f74816127a-kube-api-access-gclkb\") pod \"sub-test-model-a2ea47-kserve-6849665c99-q7tc5\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.238425 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.238363 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:23.368123 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.368098 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5"] Jul 12 00:14:23.371396 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:14:23.371368 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69a26d05_3647_43f4_a256_01f74816127a.slice/crio-fbb79ceccfdd3251323ad755eebb370b88a4cae4465480e850e82f38191256c5 WatchSource:0}: Error finding container fbb79ceccfdd3251323ad755eebb370b88a4cae4465480e850e82f38191256c5: Status 404 returned error can't find the container with id fbb79ceccfdd3251323ad755eebb370b88a4cae4465480e850e82f38191256c5 Jul 12 00:14:23.991570 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.991533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" event={"ID":"69a26d05-3647-43f4-a256-01f74816127a","Type":"ContainerStarted","Data":"f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b"} Jul 12 00:14:23.992049 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:23.991585 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" event={"ID":"69a26d05-3647-43f4-a256-01f74816127a","Type":"ContainerStarted","Data":"fbb79ceccfdd3251323ad755eebb370b88a4cae4465480e850e82f38191256c5"} Jul 12 00:14:24.008020 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:24.007974 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" podStartSLOduration=2.007961314 podStartE2EDuration="2.007961314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:14:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:14:24.006460273 +0000 UTC m=+2599.662083526" watchObservedRunningTime="2026-07-12 00:14:24.007961314 +0000 UTC m=+2599.663584566" Jul 12 00:14:33.239059 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:33.239011 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:33.239059 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:33.239071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:33.251855 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:33.251829 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:14:34.050189 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:14:34.050161 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:15:01.316001 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:01.315964 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:15:02.196319 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:02.196286 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5"] Jul 12 00:15:02.220459 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:02.220432 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:15:05.225176 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.225144 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl"] Jul 12 00:15:05.228733 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.228713 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.231605 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.231586 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"openshift-service-ca.crt\"" Jul 12 00:15:05.232902 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.232885 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"rate-test-model-a7d1ef-kserve-self-signed-certs\"" Jul 12 00:15:05.232996 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.232912 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"kube-root-ca.crt\"" Jul 12 00:15:05.233323 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.233307 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a7d1ef\"/\"default-dockercfg-7tpxz\"" Jul 12 00:15:05.237586 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.237564 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl"] Jul 12 00:15:05.244008 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.243981 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:15:05.316673 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.316649 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8nbs\" (UniqueName: \"kubernetes.io/projected/d52376ab-74d9-4938-9a77-23e4feae0c10-kube-api-access-g8nbs\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.316765 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.316700 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-home\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.316765 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.316719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-dshm\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.316765 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.316744 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52376ab-74d9-4938-9a77-23e4feae0c10-tls-certs\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.316867 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.316771 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-model-cache\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.316867 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.316804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-tmp-dir\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.417603 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.417579 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-model-cache\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.417693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.417617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-tmp-dir\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.417693 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.417644 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8nbs\" (UniqueName: \"kubernetes.io/projected/d52376ab-74d9-4938-9a77-23e4feae0c10-kube-api-access-g8nbs\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.417779 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.417726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-home\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.417779 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.417749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-dshm\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.417873 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.417775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52376ab-74d9-4938-9a77-23e4feae0c10-tls-certs\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.418005 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.417981 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-model-cache\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.418075 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.418029 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-tmp-dir\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.418139 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.418087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-home\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.419992 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.419973 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-dshm\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.420135 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.420119 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52376ab-74d9-4938-9a77-23e4feae0c10-tls-certs\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.424867 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.424846 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8nbs\" (UniqueName: \"kubernetes.io/projected/d52376ab-74d9-4938-9a77-23e4feae0c10-kube-api-access-g8nbs\") pod \"rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.537833 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.537781 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:05.669216 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:05.669194 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl"] Jul 12 00:15:05.671385 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:15:05.671360 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd52376ab_74d9_4938_9a77_23e4feae0c10.slice/crio-45bbe88c4622866c2eb3f430f3b8cde7cc6aa8454fa102b6dc1f8de652e1d3b8 WatchSource:0}: Error finding container 45bbe88c4622866c2eb3f430f3b8cde7cc6aa8454fa102b6dc1f8de652e1d3b8: Status 404 returned error can't find the container with id 45bbe88c4622866c2eb3f430f3b8cde7cc6aa8454fa102b6dc1f8de652e1d3b8 Jul 12 00:15:06.194047 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:06.194008 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" event={"ID":"d52376ab-74d9-4938-9a77-23e4feae0c10","Type":"ContainerStarted","Data":"25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8"} Jul 12 00:15:06.194240 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:06.194053 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" event={"ID":"d52376ab-74d9-4938-9a77-23e4feae0c10","Type":"ContainerStarted","Data":"45bbe88c4622866c2eb3f430f3b8cde7cc6aa8454fa102b6dc1f8de652e1d3b8"} Jul 12 00:15:06.210452 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:06.210401 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" podStartSLOduration=1.210386606 podStartE2EDuration="1.210386606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:15:06.20885484 +0000 UTC m=+2641.864478093" watchObservedRunningTime="2026-07-12 00:15:06.210386606 +0000 UTC m=+2641.866009858" Jul 12 00:15:15.538529 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:15.538486 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:15.538898 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:15.538540 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:15.551049 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:15.551019 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:16.252794 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:16.252765 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:15:17.217378 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.217341 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" podUID="69a26d05-3647-43f4-a256-01f74816127a" containerName="main" containerID="cri-o://f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b" gracePeriod=45 Jul 12 00:15:17.472097 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.472040 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:15:17.512679 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.512654 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gclkb\" (UniqueName: \"kubernetes.io/projected/69a26d05-3647-43f4-a256-01f74816127a-kube-api-access-gclkb\") pod \"69a26d05-3647-43f4-a256-01f74816127a\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " Jul 12 00:15:17.512816 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.512718 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-dshm\") pod \"69a26d05-3647-43f4-a256-01f74816127a\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " Jul 12 00:15:17.512816 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.512763 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-home\") pod \"69a26d05-3647-43f4-a256-01f74816127a\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " Jul 12 00:15:17.512816 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.512792 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a26d05-3647-43f4-a256-01f74816127a-tls-certs\") pod \"69a26d05-3647-43f4-a256-01f74816127a\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " Jul 12 00:15:17.512994 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.512838 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-tmp-dir\") pod \"69a26d05-3647-43f4-a256-01f74816127a\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " Jul 12 00:15:17.512994 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.512910 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-model-cache\") pod \"69a26d05-3647-43f4-a256-01f74816127a\" (UID: \"69a26d05-3647-43f4-a256-01f74816127a\") " Jul 12 00:15:17.513105 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.513047 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-home" (OuterVolumeSpecName: "home") pod "69a26d05-3647-43f4-a256-01f74816127a" (UID: "69a26d05-3647-43f4-a256-01f74816127a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:15:17.513250 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.513233 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:15:17.513324 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.513255 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-model-cache" (OuterVolumeSpecName: "model-cache") pod "69a26d05-3647-43f4-a256-01f74816127a" (UID: "69a26d05-3647-43f4-a256-01f74816127a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:15:17.513383 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.513352 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69a26d05-3647-43f4-a256-01f74816127a" (UID: "69a26d05-3647-43f4-a256-01f74816127a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:15:17.514889 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.514860 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-dshm" (OuterVolumeSpecName: "dshm") pod "69a26d05-3647-43f4-a256-01f74816127a" (UID: "69a26d05-3647-43f4-a256-01f74816127a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:15:17.514985 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.514942 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a26d05-3647-43f4-a256-01f74816127a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69a26d05-3647-43f4-a256-01f74816127a" (UID: "69a26d05-3647-43f4-a256-01f74816127a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:15:17.515171 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.515145 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a26d05-3647-43f4-a256-01f74816127a-kube-api-access-gclkb" (OuterVolumeSpecName: "kube-api-access-gclkb") pod "69a26d05-3647-43f4-a256-01f74816127a" (UID: "69a26d05-3647-43f4-a256-01f74816127a"). InnerVolumeSpecName "kube-api-access-gclkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:15:17.614566 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.614537 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69a26d05-3647-43f4-a256-01f74816127a-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:15:17.614566 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.614564 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:15:17.614710 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.614576 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:15:17.614710 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.614588 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gclkb\" (UniqueName: \"kubernetes.io/projected/69a26d05-3647-43f4-a256-01f74816127a-kube-api-access-gclkb\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:15:17.614710 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:17.614603 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69a26d05-3647-43f4-a256-01f74816127a-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:15:18.252247 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.252212 2566 generic.go:356] "Generic (PLEG): container finished" podID="69a26d05-3647-43f4-a256-01f74816127a" containerID="f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b" exitCode=0 Jul 12 00:15:18.252689 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.252260 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" Jul 12 00:15:18.252689 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.252299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" event={"ID":"69a26d05-3647-43f4-a256-01f74816127a","Type":"ContainerDied","Data":"f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b"} Jul 12 00:15:18.252689 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.252343 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5" event={"ID":"69a26d05-3647-43f4-a256-01f74816127a","Type":"ContainerDied","Data":"fbb79ceccfdd3251323ad755eebb370b88a4cae4465480e850e82f38191256c5"} Jul 12 00:15:18.252689 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.252361 2566 scope.go:117] "RemoveContainer" containerID="f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b" Jul 12 00:15:18.260946 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.260929 2566 scope.go:117] "RemoveContainer" containerID="f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b" Jul 12 00:15:18.261188 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:15:18.261167 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b\": container with ID starting with f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b not found: ID does not exist" containerID="f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b" Jul 12 00:15:18.261241 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.261197 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b"} err="failed to get container status \"f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b\": rpc error: code = NotFound desc = could not find container \"f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b\": container with ID starting with f8c6a6a2f5efa48505aed2f0c077ed157fffe84cb2731b759c835f97e8f7016b not found: ID does not exist" Jul 12 00:15:18.274845 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.274819 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5"] Jul 12 00:15:18.280525 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.280493 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/sub-test-model-a2ea47-kserve-6849665c99-q7tc5"] Jul 12 00:15:18.807264 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:18.807236 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a26d05-3647-43f4-a256-01f74816127a" path="/var/lib/kubelet/pods/69a26d05-3647-43f4-a256-01f74816127a/volumes" Jul 12 00:15:30.719445 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.719413 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s"] Jul 12 00:15:30.719818 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.719803 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a26d05-3647-43f4-a256-01f74816127a" containerName="main" Jul 12 00:15:30.719818 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.719819 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a26d05-3647-43f4-a256-01f74816127a" containerName="main" Jul 12 00:15:30.719911 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.719905 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a26d05-3647-43f4-a256-01f74816127a" containerName="main" Jul 12 00:15:30.724384 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.724366 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.727080 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.727060 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"openshift-service-ca.crt\"" Jul 12 00:15:30.728626 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.728594 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"kube-root-ca.crt\"" Jul 12 00:15:30.728752 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.728733 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"default-dockercfg-5w5pk\"" Jul 12 00:15:30.728806 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.728750 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a2ea47\"/\"rate-test-model-a2ea47-kserve-self-signed-certs\"" Jul 12 00:15:30.730788 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.730768 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s"] Jul 12 00:15:30.820473 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.820423 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:15:30.822928 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.822909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-model-cache\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.823005 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.822945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl9rc\" (UniqueName: \"kubernetes.io/projected/fa9fb434-8e58-43cc-8d6a-11390deaac0e-kube-api-access-jl9rc\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.823005 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.822965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-home\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.823005 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.822988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tls-certs\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.823164 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.823094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tmp-dir\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.823164 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.823150 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-dshm\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924355 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924331 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-model-cache\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924485 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924367 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl9rc\" (UniqueName: \"kubernetes.io/projected/fa9fb434-8e58-43cc-8d6a-11390deaac0e-kube-api-access-jl9rc\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924485 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924388 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-home\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924583 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924520 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tls-certs\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924641 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924621 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tmp-dir\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924715 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924696 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-dshm\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924792 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924752 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-model-cache\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.924853 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.924788 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-home\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.925119 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.925099 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tmp-dir\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.926650 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.926615 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-dshm\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.926863 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.926845 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tls-certs\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:30.931824 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:30.931805 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl9rc\" (UniqueName: \"kubernetes.io/projected/fa9fb434-8e58-43cc-8d6a-11390deaac0e-kube-api-access-jl9rc\") pod \"rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:31.035719 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:31.035664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:31.170186 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:31.170155 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s"] Jul 12 00:15:31.171900 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:15:31.171860 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa9fb434_8e58_43cc_8d6a_11390deaac0e.slice/crio-b93ac9cd1f872df3239cafdd069a35033d604faea86efb6dc9d9d7c9bf3f22cc WatchSource:0}: Error finding container b93ac9cd1f872df3239cafdd069a35033d604faea86efb6dc9d9d7c9bf3f22cc: Status 404 returned error can't find the container with id b93ac9cd1f872df3239cafdd069a35033d604faea86efb6dc9d9d7c9bf3f22cc Jul 12 00:15:31.322907 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:31.322826 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" event={"ID":"fa9fb434-8e58-43cc-8d6a-11390deaac0e","Type":"ContainerStarted","Data":"4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c"} Jul 12 00:15:31.322907 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:31.322867 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" event={"ID":"fa9fb434-8e58-43cc-8d6a-11390deaac0e","Type":"ContainerStarted","Data":"b93ac9cd1f872df3239cafdd069a35033d604faea86efb6dc9d9d7c9bf3f22cc"} Jul 12 00:15:31.360496 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:31.360450 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" podStartSLOduration=1.3604350570000001 podStartE2EDuration="1.360435057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 00:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:15:31.355007326 +0000 UTC m=+2667.010630579" watchObservedRunningTime="2026-07-12 00:15:31.360435057 +0000 UTC m=+2667.016058310" Jul 12 00:15:41.036242 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:41.036207 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:41.036706 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:41.036255 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:41.048364 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:41.048339 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:41.385823 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:41.385744 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:15:43.914110 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:43.914076 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:15:44.017355 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:44.017320 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:15:52.199475 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:52.199440 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:15:52.300608 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:15:52.300560 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:04.912829 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:04.912758 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:16:04.917128 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:04.917103 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:16:04.924720 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:04.924696 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:16:04.928732 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:04.928715 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tvxms_d77fef30-62dd-4edf-b2c9-3d14a4a1a614/ovn-acl-logging/0.log" Jul 12 00:16:13.031884 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:13.031852 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl"] Jul 12 00:16:13.055243 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:13.055216 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:13.945866 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:13.945824 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s"] Jul 12 00:16:13.962770 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:13.962742 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:28.055453 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.055416 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" podUID="d52376ab-74d9-4938-9a77-23e4feae0c10" containerName="main" containerID="cri-o://25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8" gracePeriod=45 Jul 12 00:16:28.310260 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.310200 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:16:28.395154 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395128 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-tmp-dir\") pod \"d52376ab-74d9-4938-9a77-23e4feae0c10\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " Jul 12 00:16:28.395310 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395165 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52376ab-74d9-4938-9a77-23e4feae0c10-tls-certs\") pod \"d52376ab-74d9-4938-9a77-23e4feae0c10\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " Jul 12 00:16:28.395310 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395191 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-model-cache\") pod \"d52376ab-74d9-4938-9a77-23e4feae0c10\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " Jul 12 00:16:28.395310 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395254 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8nbs\" (UniqueName: \"kubernetes.io/projected/d52376ab-74d9-4938-9a77-23e4feae0c10-kube-api-access-g8nbs\") pod \"d52376ab-74d9-4938-9a77-23e4feae0c10\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " Jul 12 00:16:28.395310 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395273 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-dshm\") pod \"d52376ab-74d9-4938-9a77-23e4feae0c10\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " Jul 12 00:16:28.395580 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395396 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-home\") pod \"d52376ab-74d9-4938-9a77-23e4feae0c10\" (UID: \"d52376ab-74d9-4938-9a77-23e4feae0c10\") " Jul 12 00:16:28.395580 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395403 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d52376ab-74d9-4938-9a77-23e4feae0c10" (UID: "d52376ab-74d9-4938-9a77-23e4feae0c10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:28.395580 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395488 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-model-cache" (OuterVolumeSpecName: "model-cache") pod "d52376ab-74d9-4938-9a77-23e4feae0c10" (UID: "d52376ab-74d9-4938-9a77-23e4feae0c10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:28.395735 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395654 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-home" (OuterVolumeSpecName: "home") pod "d52376ab-74d9-4938-9a77-23e4feae0c10" (UID: "d52376ab-74d9-4938-9a77-23e4feae0c10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:28.395795 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395781 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:28.395853 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395804 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:28.395853 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.395820 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:28.397370 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.397340 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d52376ab-74d9-4938-9a77-23e4feae0c10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d52376ab-74d9-4938-9a77-23e4feae0c10" (UID: "d52376ab-74d9-4938-9a77-23e4feae0c10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:16:28.397470 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.397373 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-dshm" (OuterVolumeSpecName: "dshm") pod "d52376ab-74d9-4938-9a77-23e4feae0c10" (UID: "d52376ab-74d9-4938-9a77-23e4feae0c10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:28.397470 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.397378 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52376ab-74d9-4938-9a77-23e4feae0c10-kube-api-access-g8nbs" (OuterVolumeSpecName: "kube-api-access-g8nbs") pod "d52376ab-74d9-4938-9a77-23e4feae0c10" (UID: "d52376ab-74d9-4938-9a77-23e4feae0c10"). InnerVolumeSpecName "kube-api-access-g8nbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:16:28.496932 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.496908 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d52376ab-74d9-4938-9a77-23e4feae0c10-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:28.496932 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.496928 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8nbs\" (UniqueName: \"kubernetes.io/projected/d52376ab-74d9-4938-9a77-23e4feae0c10-kube-api-access-g8nbs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:28.497073 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.496938 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d52376ab-74d9-4938-9a77-23e4feae0c10-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:28.613200 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.613136 2566 generic.go:356] "Generic (PLEG): container finished" podID="d52376ab-74d9-4938-9a77-23e4feae0c10" containerID="25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8" exitCode=0 Jul 12 00:16:28.613200 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.613177 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" Jul 12 00:16:28.613377 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.613218 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" event={"ID":"d52376ab-74d9-4938-9a77-23e4feae0c10","Type":"ContainerDied","Data":"25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8"} Jul 12 00:16:28.613377 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.613262 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl" event={"ID":"d52376ab-74d9-4938-9a77-23e4feae0c10","Type":"ContainerDied","Data":"45bbe88c4622866c2eb3f430f3b8cde7cc6aa8454fa102b6dc1f8de652e1d3b8"} Jul 12 00:16:28.613377 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.613279 2566 scope.go:117] "RemoveContainer" containerID="25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8" Jul 12 00:16:28.622678 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.622661 2566 scope.go:117] "RemoveContainer" containerID="25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8" Jul 12 00:16:28.622923 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:16:28.622903 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8\": container with ID starting with 25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8 not found: ID does not exist" containerID="25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8" Jul 12 00:16:28.622975 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.622931 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8"} err="failed to get container status \"25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8\": rpc error: code = NotFound desc = could not find container \"25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8\": container with ID starting with 25b049061a9d4af5af4c855d7bfd5e30c11aec1958b07f8f09f1638e6d3021c8 not found: ID does not exist" Jul 12 00:16:28.634000 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.633980 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl"] Jul 12 00:16:28.637676 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.637658 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a7d1ef/rate-test-model-a7d1ef-kserve-68cd9974f5-8nrpl"] Jul 12 00:16:28.808113 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.808089 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d52376ab-74d9-4938-9a77-23e4feae0c10" path="/var/lib/kubelet/pods/d52376ab-74d9-4938-9a77-23e4feae0c10/volumes" Jul 12 00:16:28.967569 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:28.967531 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" podUID="fa9fb434-8e58-43cc-8d6a-11390deaac0e" containerName="main" containerID="cri-o://4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c" gracePeriod=45 Jul 12 00:16:29.217527 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.217486 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:16:29.303407 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303344 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tmp-dir\") pod \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " Jul 12 00:16:29.303407 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303378 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-dshm\") pod \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " Jul 12 00:16:29.303583 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303414 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tls-certs\") pod \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " Jul 12 00:16:29.303583 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303454 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jl9rc\" (UniqueName: \"kubernetes.io/projected/fa9fb434-8e58-43cc-8d6a-11390deaac0e-kube-api-access-jl9rc\") pod \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " Jul 12 00:16:29.303583 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303539 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-model-cache\") pod \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " Jul 12 00:16:29.303583 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303571 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-home\") pod \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\" (UID: \"fa9fb434-8e58-43cc-8d6a-11390deaac0e\") " Jul 12 00:16:29.303833 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303641 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa9fb434-8e58-43cc-8d6a-11390deaac0e" (UID: "fa9fb434-8e58-43cc-8d6a-11390deaac0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:29.303833 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303807 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa9fb434-8e58-43cc-8d6a-11390deaac0e" (UID: "fa9fb434-8e58-43cc-8d6a-11390deaac0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:29.303930 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303839 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-home" (OuterVolumeSpecName: "home") pod "fa9fb434-8e58-43cc-8d6a-11390deaac0e" (UID: "fa9fb434-8e58-43cc-8d6a-11390deaac0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:29.303930 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303918 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tmp-dir\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:29.304040 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.303938 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-model-cache\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:29.305543 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.305517 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9fb434-8e58-43cc-8d6a-11390deaac0e-kube-api-access-jl9rc" (OuterVolumeSpecName: "kube-api-access-jl9rc") pod "fa9fb434-8e58-43cc-8d6a-11390deaac0e" (UID: "fa9fb434-8e58-43cc-8d6a-11390deaac0e"). InnerVolumeSpecName "kube-api-access-jl9rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:16:29.305543 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.305530 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa9fb434-8e58-43cc-8d6a-11390deaac0e" (UID: "fa9fb434-8e58-43cc-8d6a-11390deaac0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:16:29.305682 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.305645 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-dshm" (OuterVolumeSpecName: "dshm") pod "fa9fb434-8e58-43cc-8d6a-11390deaac0e" (UID: "fa9fb434-8e58-43cc-8d6a-11390deaac0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 00:16:29.404288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.404266 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-home\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:29.404288 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.404286 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa9fb434-8e58-43cc-8d6a-11390deaac0e-dshm\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:29.404425 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.404295 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa9fb434-8e58-43cc-8d6a-11390deaac0e-tls-certs\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:29.404425 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.404304 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl9rc\" (UniqueName: \"kubernetes.io/projected/fa9fb434-8e58-43cc-8d6a-11390deaac0e-kube-api-access-jl9rc\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:29.620174 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.620114 2566 generic.go:356] "Generic (PLEG): container finished" podID="fa9fb434-8e58-43cc-8d6a-11390deaac0e" containerID="4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c" exitCode=0 Jul 12 00:16:29.620174 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.620155 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" Jul 12 00:16:29.620320 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.620178 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" event={"ID":"fa9fb434-8e58-43cc-8d6a-11390deaac0e","Type":"ContainerDied","Data":"4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c"} Jul 12 00:16:29.620320 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.620208 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s" event={"ID":"fa9fb434-8e58-43cc-8d6a-11390deaac0e","Type":"ContainerDied","Data":"b93ac9cd1f872df3239cafdd069a35033d604faea86efb6dc9d9d7c9bf3f22cc"} Jul 12 00:16:29.620320 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.620225 2566 scope.go:117] "RemoveContainer" containerID="4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c" Jul 12 00:16:29.628995 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.628979 2566 scope.go:117] "RemoveContainer" containerID="4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c" Jul 12 00:16:29.629240 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:16:29.629222 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c\": container with ID starting with 4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c not found: ID does not exist" containerID="4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c" Jul 12 00:16:29.629298 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.629247 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c"} err="failed to get container status \"4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c\": rpc error: code = NotFound desc = could not find container \"4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c\": container with ID starting with 4798e3c8ee7e5f5dc1da5aa5ca43af223847dbfc0c3c7698889b026eb182e86c not found: ID does not exist" Jul 12 00:16:29.640982 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.640959 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s"] Jul 12 00:16:29.644494 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:29.644474 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a2ea47/rate-test-model-a2ea47-kserve-69c95c5fb7-68x2s"] Jul 12 00:16:30.813849 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:30.813818 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9fb434-8e58-43cc-8d6a-11390deaac0e" path="/var/lib/kubelet/pods/fa9fb434-8e58-43cc-8d6a-11390deaac0e/volumes" Jul 12 00:16:33.436854 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:33.436826 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:33.537333 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:33.537292 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:43.337468 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.337433 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j"] Jul 12 00:16:43.337931 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.337648 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" podUID="b18f9075-2c6f-4391-ab69-3a12d3658e8b" containerName="maas-api" containerID="cri-o://1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5" gracePeriod=30 Jul 12 00:16:43.357860 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.357836 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:43.589750 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.589693 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:16:43.691869 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.691841 2566 generic.go:356] "Generic (PLEG): container finished" podID="b18f9075-2c6f-4391-ab69-3a12d3658e8b" containerID="1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5" exitCode=0 Jul 12 00:16:43.692004 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.691886 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" Jul 12 00:16:43.692004 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.691932 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" event={"ID":"b18f9075-2c6f-4391-ab69-3a12d3658e8b","Type":"ContainerDied","Data":"1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5"} Jul 12 00:16:43.692004 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.691986 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j" event={"ID":"b18f9075-2c6f-4391-ab69-3a12d3658e8b","Type":"ContainerDied","Data":"542f1c6b4b4b871a9f9efcf05554ad917e3e1b63a96e3f23f35da89a2cf5e76c"} Jul 12 00:16:43.692140 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.692015 2566 scope.go:117] "RemoveContainer" containerID="1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5" Jul 12 00:16:43.700148 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.700123 2566 scope.go:117] "RemoveContainer" containerID="1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5" Jul 12 00:16:43.700396 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:16:43.700378 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5\": container with ID starting with 1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5 not found: ID does not exist" containerID="1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5" Jul 12 00:16:43.700447 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.700406 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5"} err="failed to get container status \"1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5\": rpc error: code = NotFound desc = could not find container \"1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5\": container with ID starting with 1a0d93f9ed13228f6c436255b6babb28d8336b33bf092a7c4a5400591ded47f5 not found: ID does not exist" Jul 12 00:16:43.723683 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.723666 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzpwq\" (UniqueName: \"kubernetes.io/projected/b18f9075-2c6f-4391-ab69-3a12d3658e8b-kube-api-access-mzpwq\") pod \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " Jul 12 00:16:43.723762 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.723709 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f9075-2c6f-4391-ab69-3a12d3658e8b-openshift-service-ca\") pod \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " Jul 12 00:16:43.723762 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.723729 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls\") pod \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\" (UID: \"b18f9075-2c6f-4391-ab69-3a12d3658e8b\") " Jul 12 00:16:43.724042 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.724019 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18f9075-2c6f-4391-ab69-3a12d3658e8b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b18f9075-2c6f-4391-ab69-3a12d3658e8b" (UID: "b18f9075-2c6f-4391-ab69-3a12d3658e8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 00:16:43.725637 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.725617 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b18f9075-2c6f-4391-ab69-3a12d3658e8b" (UID: "b18f9075-2c6f-4391-ab69-3a12d3658e8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:16:43.725704 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.725690 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18f9075-2c6f-4391-ab69-3a12d3658e8b-kube-api-access-mzpwq" (OuterVolumeSpecName: "kube-api-access-mzpwq") pod "b18f9075-2c6f-4391-ab69-3a12d3658e8b" (UID: "b18f9075-2c6f-4391-ab69-3a12d3658e8b"). InnerVolumeSpecName "kube-api-access-mzpwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:16:43.824803 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.824780 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzpwq\" (UniqueName: \"kubernetes.io/projected/b18f9075-2c6f-4391-ab69-3a12d3658e8b-kube-api-access-mzpwq\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:43.824803 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.824801 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b18f9075-2c6f-4391-ab69-3a12d3658e8b-openshift-service-ca\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:43.824932 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:43.824810 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b18f9075-2c6f-4391-ab69-3a12d3658e8b-maas-api-tls\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:16:44.013602 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:44.013575 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j"] Jul 12 00:16:44.015480 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:44.015459 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a7d1ef-6d86c86b69-88v2j"] Jul 12 00:16:44.807009 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:44.806977 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18f9075-2c6f-4391-ab69-3a12d3658e8b" path="/var/lib/kubelet/pods/b18f9075-2c6f-4391-ab69-3a12d3658e8b/volumes" Jul 12 00:16:48.065054 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:48.065022 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:52.572926 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:52.572895 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:52.623007 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:52.622977 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:53.834875 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:53.834843 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:53.892175 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:53.892143 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:58.724674 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:58.724647 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:58.948229 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:58.948191 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:59.543670 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:59.543639 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:16:59.642551 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:16:59.642494 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:09.426605 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.426570 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp"] Jul 12 00:17:09.427069 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.426846 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" podUID="c3b7bfec-d6d8-4b41-8059-d05efe4cd435" containerName="maas-api" containerID="cri-o://0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4" gracePeriod=30 Jul 12 00:17:09.453077 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.453055 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:09.669276 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.669252 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:17:09.730764 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.730692 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64kb9\" (UniqueName: \"kubernetes.io/projected/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-kube-api-access-64kb9\") pod \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " Jul 12 00:17:09.730764 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.730728 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls\") pod \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " Jul 12 00:17:09.730954 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.730770 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-openshift-service-ca\") pod \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\" (UID: \"c3b7bfec-d6d8-4b41-8059-d05efe4cd435\") " Jul 12 00:17:09.731199 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.731167 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3b7bfec-d6d8-4b41-8059-d05efe4cd435" (UID: "c3b7bfec-d6d8-4b41-8059-d05efe4cd435"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 00:17:09.732785 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.732757 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3b7bfec-d6d8-4b41-8059-d05efe4cd435" (UID: "c3b7bfec-d6d8-4b41-8059-d05efe4cd435"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 00:17:09.732785 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.732771 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-kube-api-access-64kb9" (OuterVolumeSpecName: "kube-api-access-64kb9") pod "c3b7bfec-d6d8-4b41-8059-d05efe4cd435" (UID: "c3b7bfec-d6d8-4b41-8059-d05efe4cd435"). InnerVolumeSpecName "kube-api-access-64kb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 00:17:09.816558 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.816533 2566 generic.go:356] "Generic (PLEG): container finished" podID="c3b7bfec-d6d8-4b41-8059-d05efe4cd435" containerID="0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4" exitCode=0 Jul 12 00:17:09.816681 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.816640 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" Jul 12 00:17:09.816681 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.816654 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" event={"ID":"c3b7bfec-d6d8-4b41-8059-d05efe4cd435","Type":"ContainerDied","Data":"0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4"} Jul 12 00:17:09.816770 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.816690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp" event={"ID":"c3b7bfec-d6d8-4b41-8059-d05efe4cd435","Type":"ContainerDied","Data":"146725b7272aea3512f9a5011e3453629b881a3353b6c00f33894797dc3618b4"} Jul 12 00:17:09.816770 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.816712 2566 scope.go:117] "RemoveContainer" containerID="0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4" Jul 12 00:17:09.825481 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.825463 2566 scope.go:117] "RemoveContainer" containerID="0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4" Jul 12 00:17:09.825731 ip-10-0-137-251 kubenswrapper[2566]: E0712 00:17:09.825713 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4\": container with ID starting with 0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4 not found: ID does not exist" containerID="0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4" Jul 12 00:17:09.825794 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.825738 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4"} err="failed to get container status \"0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4\": rpc error: code = NotFound desc = could not find container \"0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4\": container with ID starting with 0cf108efd0e796d3792e24bd37fc5ea5e960d8cc93abed84af8a1a18fc7a3ef4 not found: ID does not exist" Jul 12 00:17:09.831928 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.831908 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64kb9\" (UniqueName: \"kubernetes.io/projected/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-kube-api-access-64kb9\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:17:09.831928 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.831929 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-maas-api-tls\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:17:09.832049 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.831938 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b7bfec-d6d8-4b41-8059-d05efe4cd435-openshift-service-ca\") on node \"ip-10-0-137-251.ec2.internal\" DevicePath \"\"" Jul 12 00:17:09.836643 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.836622 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp"] Jul 12 00:17:09.840251 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:09.840231 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-a2ea47-655db8968b-rbjkp"] Jul 12 00:17:10.807563 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:10.807534 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3b7bfec-d6d8-4b41-8059-d05efe4cd435" path="/var/lib/kubelet/pods/c3b7bfec-d6d8-4b41-8059-d05efe4cd435/volumes" Jul 12 00:17:14.150994 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:14.150961 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:19.984735 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:19.984690 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:20.071921 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:20.071893 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:24.847179 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:24.847150 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:25.077597 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:25.077552 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:27.228491 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:27.228399 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:27.278250 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:27.278221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:31.040552 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:31.040518 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7fv2f_9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1/server/0.log" Jul 12 00:17:31.158186 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:31.158158 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t9jqq_8e0719d0-9b02-4ded-9537-800d494953a4/manager/0.log" Jul 12 00:17:31.392695 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:31.392675 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d55578dd8-4wqm8_63a38ae4-0a39-455b-88a9-02d6c71cf4fe/copy-manifests/0.log" Jul 12 00:17:31.404000 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:31.403975 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d55578dd8-4wqm8_63a38ae4-0a39-455b-88a9-02d6c71cf4fe/manager/0.log" Jul 12 00:17:33.757099 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:33.757057 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-mkmk9_0ae682fd-6f3f-43df-9e20-4e014a9e757a/postgres/0.log" Jul 12 00:17:35.212835 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:35.212805 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-jclsm_bd1007f2-e8ed-453c-92a2-36c5d0671354/manager/0.log" Jul 12 00:17:35.473116 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:35.473037 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:35.565381 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:35.565353 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xh2gs_9abac196-4ee5-4fb9-b41e-01d800152418/registry-server/0.log" Jul 12 00:17:35.693372 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:35.693350 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vpv4f_45e054f7-67f1-4acf-a65a-0569e16f8105/manager/0.log" Jul 12 00:17:35.953727 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:35.953706 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kkqmf_f4c3cd5e-1449-4aae-88bf-234df0ed8e44/manager/0.log" Jul 12 00:17:37.146407 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:37.146382 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7bdd5f5b-nw2lj_67d5e183-87ab-406e-a885-243f0e2eff99/router/0.log" Jul 12 00:17:37.733097 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:37.733057 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-s99pc_aaa1332a-bea2-46b6-9df4-caa60d58f8e8/main/0.log" Jul 12 00:17:37.959694 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:37.959669 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-nqq9s_e6590983-dbf5-4888-b2d3-d135540ff2d2/main/0.log" Jul 12 00:17:38.070431 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:38.070362 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-6gxbz_1b8da3b6-2929-4133-857b-b887548a8d52/main/0.log" Jul 12 00:17:42.223406 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:42.223376 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:42.322895 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:42.322867 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" limit=1024 Jul 12 00:17:45.131191 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:45.131167 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q5tgg_80391286-a001-4714-95f7-47d673ca1a6b/global-pull-secret-syncer/0.log" Jul 12 00:17:45.174292 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:45.174272 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qbjd9_d87e378d-6b17-43a7-8667-e095db0b08b0/konnectivity-agent/0.log" Jul 12 00:17:45.274464 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:45.274442 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-251.ec2.internal_554e1624a4d584e0be3ea6067bec89c5/haproxy/0.log" Jul 12 00:17:50.235991 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:50.235964 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-jclsm_bd1007f2-e8ed-453c-92a2-36c5d0671354/manager/0.log" Jul 12 00:17:50.325750 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:50.325724 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xh2gs_9abac196-4ee5-4fb9-b41e-01d800152418/registry-server/0.log" Jul 12 00:17:50.446058 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:50.446024 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-vpv4f_45e054f7-67f1-4acf-a65a-0569e16f8105/manager/0.log" Jul 12 00:17:50.563952 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:50.563881 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-kkqmf_f4c3cd5e-1449-4aae-88bf-234df0ed8e44/manager/0.log" Jul 12 00:17:52.402826 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:52.402797 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-9qnk4_ca6e6f9c-e4b4-4b6f-8fc8-b04def114b80/cluster-monitoring-operator/0.log" Jul 12 00:17:52.579056 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:52.578999 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wq7w_b23100f6-9023-49d8-b503-a05c08a0c73b/node-exporter/0.log" Jul 12 00:17:52.599425 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:52.599405 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wq7w_b23100f6-9023-49d8-b503-a05c08a0c73b/kube-rbac-proxy/0.log" Jul 12 00:17:52.626101 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:52.626085 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wq7w_b23100f6-9023-49d8-b503-a05c08a0c73b/init-textfile/0.log" Jul 12 00:17:52.975059 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:52.975033 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11b50933-4fa0-433d-8b27-f2f83f2abe59/prometheus/0.log" Jul 12 00:17:52.999207 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:52.999190 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11b50933-4fa0-433d-8b27-f2f83f2abe59/config-reloader/0.log" Jul 12 00:17:53.022892 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:53.022875 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11b50933-4fa0-433d-8b27-f2f83f2abe59/thanos-sidecar/0.log" Jul 12 00:17:53.045198 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:53.045181 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11b50933-4fa0-433d-8b27-f2f83f2abe59/kube-rbac-proxy-web/0.log" Jul 12 00:17:53.069689 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:53.069673 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11b50933-4fa0-433d-8b27-f2f83f2abe59/kube-rbac-proxy/0.log" Jul 12 00:17:53.096361 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:53.096344 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11b50933-4fa0-433d-8b27-f2f83f2abe59/kube-rbac-proxy-thanos/0.log" Jul 12 00:17:53.118976 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:53.118953 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_11b50933-4fa0-433d-8b27-f2f83f2abe59/init-config-reloader/0.log" Jul 12 00:17:54.639966 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:54.639939 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-rdf9q_05d45c15-52aa-4126-8ec5-3ab300906ce5/networking-console-plugin/0.log" Jul 12 00:17:55.247780 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.247751 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/1.log" Jul 12 00:17:55.252317 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.252295 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-v692p_fb879140-59b4-4467-adfe-460df17e72be/console-operator/2.log" Jul 12 00:17:55.522756 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.522685 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp"] Jul 12 00:17:55.523041 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523029 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18f9075-2c6f-4391-ab69-3a12d3658e8b" containerName="maas-api" Jul 12 00:17:55.523086 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523042 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18f9075-2c6f-4391-ab69-3a12d3658e8b" containerName="maas-api" Jul 12 00:17:55.523086 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523066 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d52376ab-74d9-4938-9a77-23e4feae0c10" containerName="main" Jul 12 00:17:55.523086 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523072 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d52376ab-74d9-4938-9a77-23e4feae0c10" containerName="main" Jul 12 00:17:55.523086 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523084 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b7bfec-d6d8-4b41-8059-d05efe4cd435" containerName="maas-api" Jul 12 00:17:55.523208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523090 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b7bfec-d6d8-4b41-8059-d05efe4cd435" containerName="maas-api" Jul 12 00:17:55.523208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523097 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9fb434-8e58-43cc-8d6a-11390deaac0e" containerName="main" Jul 12 00:17:55.523208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523102 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9fb434-8e58-43cc-8d6a-11390deaac0e" containerName="main" Jul 12 00:17:55.523208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523152 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9fb434-8e58-43cc-8d6a-11390deaac0e" containerName="main" Jul 12 00:17:55.523208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523162 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b7bfec-d6d8-4b41-8059-d05efe4cd435" containerName="maas-api" Jul 12 00:17:55.523208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523169 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18f9075-2c6f-4391-ab69-3a12d3658e8b" containerName="maas-api" Jul 12 00:17:55.523208 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.523175 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d52376ab-74d9-4938-9a77-23e4feae0c10" containerName="main" Jul 12 00:17:55.526602 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.526582 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.529447 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.529426 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-scjg4\"/\"default-dockercfg-hhxcj\"" Jul 12 00:17:55.529575 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.529456 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scjg4\"/\"kube-root-ca.crt\"" Jul 12 00:17:55.529575 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.529427 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-scjg4\"/\"openshift-service-ca.crt\"" Jul 12 00:17:55.533440 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.533416 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp"] Jul 12 00:17:55.692120 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.692090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-proc\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.692453 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.692172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-lib-modules\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.692453 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.692200 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-podres\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.692453 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.692227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-sys\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.692453 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.692274 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7kzq\" (UniqueName: \"kubernetes.io/projected/7910c584-a090-4f67-aa6b-9c13571dd400-kube-api-access-r7kzq\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.792789 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792730 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-proc\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.792899 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792799 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-lib-modules\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.792899 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792832 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-podres\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.792899 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792851 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-proc\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.792899 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792855 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-sys\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.793117 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7kzq\" (UniqueName: \"kubernetes.io/projected/7910c584-a090-4f67-aa6b-9c13571dd400-kube-api-access-r7kzq\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.793117 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792907 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-sys\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.793117 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792961 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-podres\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.793117 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.792971 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7910c584-a090-4f67-aa6b-9c13571dd400-lib-modules\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.800583 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.800560 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7kzq\" (UniqueName: \"kubernetes.io/projected/7910c584-a090-4f67-aa6b-9c13571dd400-kube-api-access-r7kzq\") pod \"perf-node-gather-daemonset-jthbp\" (UID: \"7910c584-a090-4f67-aa6b-9c13571dd400\") " pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.837311 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.837291 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:55.869769 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.869743 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-596nr_bb92ba2e-4451-4c21-8bb1-951bdd1b9079/download-server/0.log" Jul 12 00:17:55.971035 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:55.971013 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp"] Jul 12 00:17:55.973165 ip-10-0-137-251 kubenswrapper[2566]: W0712 00:17:55.973136 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7910c584_a090_4f67_aa6b_9c13571dd400.slice/crio-a68635f1db475d9357f6754b84ced6d98ae5f7e033d6aee5b7fae98e2f6bab80 WatchSource:0}: Error finding container a68635f1db475d9357f6754b84ced6d98ae5f7e033d6aee5b7fae98e2f6bab80: Status 404 returned error can't find the container with id a68635f1db475d9357f6754b84ced6d98ae5f7e033d6aee5b7fae98e2f6bab80 Jul 12 00:17:56.031209 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:56.031183 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" event={"ID":"7910c584-a090-4f67-aa6b-9c13571dd400","Type":"ContainerStarted","Data":"a68635f1db475d9357f6754b84ced6d98ae5f7e033d6aee5b7fae98e2f6bab80"} Jul 12 00:17:56.492956 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:56.492931 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-n7hdx_1edd63df-f7bf-42b1-8384-a5a4d08232ef/volume-data-source-validator/0.log" Jul 12 00:17:57.037379 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:57.037352 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" event={"ID":"7910c584-a090-4f67-aa6b-9c13571dd400","Type":"ContainerStarted","Data":"1276756de78b1ba7cfa635d2c3dd52f2e97620541a311298002b6616aa4db6db"} Jul 12 00:17:57.037760 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:57.037464 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:17:57.052961 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:57.052920 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" podStartSLOduration=2.052906654 podStartE2EDuration="2.052906654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 00:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 00:17:57.051544525 +0000 UTC m=+2812.707167802" watchObservedRunningTime="2026-07-12 00:17:57.052906654 +0000 UTC m=+2812.708529906" Jul 12 00:17:57.505336 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:57.505315 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7tlrm_f1db19fb-9d59-4530-b624-e45f5590ff47/dns/0.log" Jul 12 00:17:57.526662 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:57.526641 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7tlrm_f1db19fb-9d59-4530-b624-e45f5590ff47/kube-rbac-proxy/0.log" Jul 12 00:17:57.665939 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:57.665912 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-f58r4_67a00afc-3605-4f51-9303-12299c30fd84/dns-node-resolver/0.log" Jul 12 00:17:58.263591 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:58.263562 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lwcsq_95d3c2ae-d983-46de-a4fd-a417d0059a37/node-ca/0.log" Jul 12 00:17:59.535598 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:17:59.535574 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7bdd5f5b-nw2lj_67d5e183-87ab-406e-a885-243f0e2eff99/router/0.log" Jul 12 00:18:00.259906 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:00.259880 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kqrpg_fce62751-789d-4923-8302-a07368e66a11/serve-healthcheck-canary/0.log" Jul 12 00:18:00.726732 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:00.726702 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-4qd7j_6057afd9-b8a3-4f14-9f90-af1beeba6e3a/insights-operator/0.log" Jul 12 00:18:00.727306 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:00.727289 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-4qd7j_6057afd9-b8a3-4f14-9f90-af1beeba6e3a/insights-operator/1.log" Jul 12 00:18:00.747999 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:00.747976 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2l9dx_d7c0b6ae-142a-40a8-ab9b-c56878988bed/kube-rbac-proxy/0.log" Jul 12 00:18:00.770878 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:00.770860 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2l9dx_d7c0b6ae-142a-40a8-ab9b-c56878988bed/exporter/0.log" Jul 12 00:18:00.791784 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:00.791766 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2l9dx_d7c0b6ae-142a-40a8-ab9b-c56878988bed/extractor/0.log" Jul 12 00:18:03.052455 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:03.052428 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-scjg4/perf-node-gather-daemonset-jthbp" Jul 12 00:18:03.241901 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:03.241878 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7fv2f_9dbde056-e4d7-41aa-9c0b-9f5c206e0ff1/server/0.log" Jul 12 00:18:03.305238 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:03.305180 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-t9jqq_8e0719d0-9b02-4ded-9537-800d494953a4/manager/0.log" Jul 12 00:18:03.425842 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:03.425816 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d55578dd8-4wqm8_63a38ae4-0a39-455b-88a9-02d6c71cf4fe/manager/0.log" Jul 12 00:18:03.443063 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:03.443037 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d55578dd8-4wqm8_63a38ae4-0a39-455b-88a9-02d6c71cf4fe/copy-manifests/0.log" Jul 12 00:18:04.849115 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:04.849088 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59d758c946-qfznb_983e865e-4d67-41ea-9def-fa43dc2dac79/manager/0.log" Jul 12 00:18:05.941279 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:05.941247 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7bdd5f5b-nw2lj_67d5e183-87ab-406e-a885-243f0e2eff99/router/0.log" Jul 12 00:18:07.408512 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:07.408473 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-c7bdd5f5b-nw2lj_67d5e183-87ab-406e-a885-243f0e2eff99/router/0.log" Jul 12 00:18:10.292181 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:10.292134 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zvwpm_22088323-b6b3-4e92-bc92-7b44ad14c8dc/kube-storage-version-migrator-operator/1.log" Jul 12 00:18:10.293028 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:10.293010 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-zvwpm_22088323-b6b3-4e92-bc92-7b44ad14c8dc/kube-storage-version-migrator-operator/0.log" Jul 12 00:18:11.253963 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.253931 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzbk4_d08914c3-1c8f-4788-9b12-3cb52cb04ba9/kube-multus-additional-cni-plugins/0.log" Jul 12 00:18:11.273679 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.273657 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzbk4_d08914c3-1c8f-4788-9b12-3cb52cb04ba9/egress-router-binary-copy/0.log" Jul 12 00:18:11.293212 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.293189 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzbk4_d08914c3-1c8f-4788-9b12-3cb52cb04ba9/cni-plugins/0.log" Jul 12 00:18:11.312377 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.312357 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzbk4_d08914c3-1c8f-4788-9b12-3cb52cb04ba9/bond-cni-plugin/0.log" Jul 12 00:18:11.332359 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.332339 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzbk4_d08914c3-1c8f-4788-9b12-3cb52cb04ba9/routeoverride-cni/0.log" Jul 12 00:18:11.352060 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.352043 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzbk4_d08914c3-1c8f-4788-9b12-3cb52cb04ba9/whereabouts-cni-bincopy/0.log" Jul 12 00:18:11.371974 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.371958 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gzbk4_d08914c3-1c8f-4788-9b12-3cb52cb04ba9/whereabouts-cni/0.log" Jul 12 00:18:11.754388 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.754366 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-phdpk_941ffd9d-f5b2-4c5c-837b-e8bf8e26481b/kube-multus/0.log" Jul 12 00:18:11.775959 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.775935 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2jq6z_67e8fbc2-1ffc-42d1-8ea6-307da940eda6/network-metrics-daemon/0.log" Jul 12 00:18:11.795356 ip-10-0-137-251 kubenswrapper[2566]: I0712 00:18:11.795335 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-2jq6z_67e8fbc2-1ffc-42d1-8ea6-307da940eda6/kube-rbac-proxy/0.log"