Jul 23 15:00:12.312269 ip-10-0-139-2 systemd[1]: Starting Kubernetes Kubelet... Jul 23 15:00:12.632669 ip-10-0-139-2 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 15:00:12.632669 ip-10-0-139-2 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 15:00:12.632669 ip-10-0-139-2 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 15:00:12.632669 ip-10-0-139-2 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 15:00:12.632669 ip-10-0-139-2 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 15:00:12.632669 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.614144 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720431 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720447 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720452 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720459 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720463 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720466 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720469 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720471 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720474 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720477 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720479 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720482 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720485 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720487 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720490 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720493 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720495 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720498 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720501 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 15:00:12.745892 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720504 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720507 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720510 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720513 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720517 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720522 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720526 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720531 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720535 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720538 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720540 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720543 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720545 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720548 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720551 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720554 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720556 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720559 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720562 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 15:00:12.746496 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720564 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720567 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720569 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720572 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720574 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720578 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720582 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720585 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720588 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720590 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720593 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720596 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720600 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720602 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720605 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720608 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720610 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720613 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720615 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720618 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 15:00:12.747343 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720620 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720623 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720627 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720630 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720633 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720636 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720639 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720642 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720645 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720648 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720650 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720653 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720655 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720658 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720661 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720663 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720666 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720668 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720671 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720673 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 15:00:12.748160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720676 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720681 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720685 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720689 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720691 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720695 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720698 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720701 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720703 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720706 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720708 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720711 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720714 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720716 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720718 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720721 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720724 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720727 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720729 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 15:00:12.748771 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720732 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.720735 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721469 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721477 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721480 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721483 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721485 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721488 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721491 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721493 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721496 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721498 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721500 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721503 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721506 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721511 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721515 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721518 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721521 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 15:00:12.749323 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721525 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721528 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721530 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721533 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721535 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721538 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721541 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721543 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721546 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721548 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721551 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721555 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721559 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721563 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721566 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721569 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721572 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721574 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721577 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 15:00:12.749919 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721580 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721582 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721585 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721587 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721590 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721592 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721595 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721597 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721600 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721602 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721605 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721607 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721610 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721613 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721616 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721618 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721621 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721624 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721626 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721629 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 15:00:12.750444 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721634 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721637 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721640 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721643 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721646 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721649 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721652 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721655 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721658 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721661 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721664 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721666 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721669 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721671 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721674 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721676 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721679 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721682 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721684 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721686 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 15:00:12.751160 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721689 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721692 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721696 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721699 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721701 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721704 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721707 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721710 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721713 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721715 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721718 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721721 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721723 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721726 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721728 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721730 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721733 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721735 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721738 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721740 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 15:00:12.751737 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721743 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721746 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.721748 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722687 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722695 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722700 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722704 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722707 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722710 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722713 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722716 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722719 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722721 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722724 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722728 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722731 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722734 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722737 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722739 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 15:00:12.752359 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722743 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722747 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722750 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722752 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722755 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722758 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722760 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722763 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722765 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722768 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722771 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722774 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722776 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722780 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722783 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722785 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722788 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722791 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722793 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722796 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 15:00:12.752863 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722799 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722801 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722804 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722807 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722809 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722812 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722814 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722816 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722819 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722823 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722826 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722834 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722836 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722841 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722844 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722847 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722849 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722852 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722854 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 15:00:12.753656 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722857 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722859 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722862 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722864 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722867 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722869 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722872 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722875 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722877 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722880 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722882 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722885 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722887 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722890 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722892 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722895 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722897 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722900 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722902 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722905 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 15:00:12.754310 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722908 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722911 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722913 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722916 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722919 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722922 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722925 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722929 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722932 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722934 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722937 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722939 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722942 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722944 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722947 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722949 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722952 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722954 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722957 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722959 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 15:00:12.754970 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722962 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722964 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722967 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.722969 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723084 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723092 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723099 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723109 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723115 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723120 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723126 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723131 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723134 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723137 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723140 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723144 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723148 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723151 2568 flags.go:64] FLAG: --cgroup-root="" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723155 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723158 2568 flags.go:64] FLAG: --client-ca-file="" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723162 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723166 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723170 2568 flags.go:64] FLAG: --cluster-domain="" Jul 23 15:00:12.755645 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723173 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723176 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723179 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723182 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723186 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723189 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723192 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723195 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723198 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723201 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723204 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723207 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723210 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723214 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723217 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723220 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723223 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723226 2568 flags.go:64] FLAG: --enable-server="true" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723228 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723232 2568 flags.go:64] FLAG: --event-burst="100" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723235 2568 flags.go:64] FLAG: --event-qps="50" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723238 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723241 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723244 2568 flags.go:64] FLAG: --eviction-hard="" Jul 23 15:00:12.756479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723248 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723252 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723255 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723259 2568 flags.go:64] FLAG: --eviction-soft="" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723262 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723264 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723268 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723271 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723274 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723277 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723279 2568 flags.go:64] FLAG: --feature-gates="" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723283 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723286 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723289 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723293 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723296 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723299 2568 flags.go:64] FLAG: --help="false" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723301 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-139-2.ec2.internal" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723304 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723307 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723310 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723313 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723316 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723319 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 15:00:12.757301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723322 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723324 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723327 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723330 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723333 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723335 2568 flags.go:64] FLAG: --kube-reserved="" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723338 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723341 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723344 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723348 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723352 2568 flags.go:64] FLAG: --lock-file="" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723356 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723359 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723362 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723367 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723370 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723373 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723376 2568 flags.go:64] FLAG: --logging-format="text" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723379 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723382 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723385 2568 flags.go:64] FLAG: --manifest-url="" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723388 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723392 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723395 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723398 2568 flags.go:64] FLAG: --max-pods="110" Jul 23 15:00:12.758335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723401 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723404 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723407 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723410 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723413 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723416 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723418 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723425 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723428 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723431 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723434 2568 flags.go:64] FLAG: --pod-cidr="" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723436 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723442 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723445 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723448 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723451 2568 flags.go:64] FLAG: --port="10250" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723454 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723457 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08efcc101009132c5" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723462 2568 flags.go:64] FLAG: --qos-reserved="" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723465 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723468 2568 flags.go:64] FLAG: --register-node="true" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723471 2568 flags.go:64] FLAG: --register-with-taints="" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723474 2568 flags.go:64] FLAG: --registry-burst="10" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723478 2568 flags.go:64] FLAG: --registry-qps="5" Jul 23 15:00:12.761008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723481 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723483 2568 flags.go:64] FLAG: --reserved-memory="" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723487 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723490 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723493 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723496 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723499 2568 flags.go:64] FLAG: --runonce="false" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723502 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723505 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723508 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723510 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723513 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723517 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723520 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723523 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723525 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723528 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723531 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723533 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723536 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723539 2568 flags.go:64] FLAG: --system-cgroups="" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723542 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723547 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723550 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723553 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 15:00:12.850018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723559 2568 flags.go:64] FLAG: --tls-min-version="" Jul 23 15:00:12.801702 ip-10-0-139-2 systemd[1]: Started Kubernetes Kubelet. Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723563 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723566 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723569 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723572 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723575 2568 flags.go:64] FLAG: --v="2" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723579 2568 flags.go:64] FLAG: --version="false" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723584 2568 flags.go:64] FLAG: --vmodule="" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723588 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.723591 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723713 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723717 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723720 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723723 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723725 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723730 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723733 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723737 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723740 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723743 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723746 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 15:00:12.852019 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723749 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723752 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723754 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723757 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723760 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723763 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723765 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723768 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723770 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723773 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723775 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723779 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723783 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723786 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723789 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723791 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723794 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723797 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723799 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723802 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 15:00:12.853035 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723805 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723808 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723810 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723813 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723817 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723820 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723823 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723826 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723829 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723832 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723835 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723838 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723841 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723844 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723846 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723849 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723851 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723854 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723857 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 15:00:12.853746 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723860 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723862 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723864 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723867 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723872 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723876 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723879 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723881 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723885 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723888 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723891 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723893 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723896 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723899 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723901 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723904 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723906 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723909 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723911 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723914 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 15:00:12.854556 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723916 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723919 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723921 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723924 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723927 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723929 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723932 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723934 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723937 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723940 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723942 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723945 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723948 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723951 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723953 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723956 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723959 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723963 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723966 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 15:00:12.855263 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723968 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723971 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723974 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723976 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723979 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723981 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723984 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.723987 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.724006 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.724010 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.724631 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.730911 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.730926 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.731502 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 15:00:12.856103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.731509 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731570 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731575 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731578 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731581 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731584 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731588 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731590 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731593 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731597 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731601 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731604 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731607 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731610 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731612 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731615 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731618 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731621 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731624 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731627 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 15:00:12.856657 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731629 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731632 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731636 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731640 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731643 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731647 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731649 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731653 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731655 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731658 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731661 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731664 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731667 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731669 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731672 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731674 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731677 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731680 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731683 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 15:00:12.857342 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731688 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731691 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731693 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731696 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731699 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731701 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731704 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731707 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731709 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731712 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731715 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731718 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731721 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731723 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731726 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731728 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731731 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731733 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731736 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731738 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 15:00:12.858138 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731741 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731743 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731747 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731750 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731754 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731757 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731760 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731763 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731765 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731768 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731770 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731773 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731777 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731782 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731785 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731787 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731790 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731792 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731795 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731797 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 15:00:12.858924 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731800 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731803 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731806 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731808 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731811 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731813 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731816 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731819 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731821 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731823 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731826 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731828 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731831 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731833 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731836 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731839 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731841 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731844 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731847 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731850 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 15:00:12.859514 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731852 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.731857 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.731986 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732010 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732016 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732021 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732025 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732028 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732032 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732036 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732040 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732043 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732046 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732048 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 15:00:12.978643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732051 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732054 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732057 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732060 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732062 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732065 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732068 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732070 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732073 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732076 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732078 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732081 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732083 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732086 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732089 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732092 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732096 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732098 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732101 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 15:00:13.004410 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732104 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732106 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732109 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732113 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732116 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732119 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732122 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732124 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732127 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732130 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732132 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732135 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732137 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732140 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732142 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732145 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732147 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732150 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732152 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732155 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 15:00:13.005459 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732157 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732160 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732162 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732165 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732167 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732174 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732177 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732179 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732182 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732184 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732193 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732196 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732199 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732201 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732204 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732208 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732210 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732215 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732217 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732220 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 15:00:13.006491 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732222 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732225 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732227 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732231 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732234 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732237 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732240 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732243 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732246 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732248 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732252 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732255 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732257 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732260 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732263 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732265 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732268 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732270 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732273 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 15:00:13.007299 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732275 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732278 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732280 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732283 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732291 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732294 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732296 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732299 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:12.732303 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.732308 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.732430 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.734405 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.735519 2568 server.go:1019] "Starting client certificate rotation" Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.735638 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 15:00:13.007893 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.735674 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 15:00:13.008533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.737935 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 15:00:13.008533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.748066 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 15:00:13.008533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.768305 2568 log.go:25] "Validated CRI v1 runtime API" Jul 23 15:00:13.008533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.770252 2568 log.go:25] "Validated CRI v1 image API" Jul 23 15:00:13.008533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.770891 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 15:00:13.008533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.777029 2568 fs.go:135] Filesystem UUIDs: map[3362a139-08a9-4f4e-9788-95e6457689c1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fe64a2f7-eccd-4515-8d47-d23363ce5127:/dev/nvme0n1p4] Jul 23 15:00:13.008533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.777045 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.782337 2568 manager.go:216] Machine: {Timestamp:2026-07-23 15:00:12.781319754 +0000 UTC m=+0.374770666 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099282 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274123afead3713cb52d0b1d6f354c SystemUUID:ec274123-afea-d371-3cb5-2d0b1d6f354c BootID:ada2f5cc-d32a-4e42-8dc4-0e8455c39e8a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:01:3f:30:d8:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:01:3f:30:d8:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:12:52:4e:8b:f3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.782431 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.782528 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.784829 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.785002 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.785040 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-2.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.785616 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.785626 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.785633 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.786170 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.787289 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791140 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791261 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791282 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791291 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791303 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 15:00:13.009629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791311 2568 kubelet.go:398] "Adding apiserver pod source" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791319 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791590 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.791604 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.794388 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.795972 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.796043 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797038 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797050 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797057 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797062 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797070 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797076 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797084 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797089 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797097 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797102 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797110 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.797115 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.798285 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.798293 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.801028 2568 server.go:1268] "Started kubelet" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.801110 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.801160 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.801189 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.814493 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.816364 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.818821 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-2.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.818949 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.818976 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.821286 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.821294 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.821472 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831056 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831058 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831088 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.818983 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-2.ec2.internal.18c4f340fd7039eb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-2.ec2.internal,UID:ip-10-0-139-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-2.ec2.internal,},FirstTimestamp:2026-07-23 15:00:12.801006059 +0000 UTC m=+0.394456969,LastTimestamp:2026-07-23 15:00:12.801006059 +0000 UTC m=+0.394456969,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-2.ec2.internal,}" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831156 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831173 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.831453 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831581 2568 factory.go:55] Registering systemd factory Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831597 2568 factory.go:223] Registration of the systemd container factory successfully Jul 23 15:00:13.010540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831797 2568 factory.go:153] Registering CRI-O factory Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831808 2568 factory.go:223] Registration of the crio container factory successfully Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831870 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831900 2568 factory.go:103] Registering Raw factory Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.831911 2568 manager.go:1196] Started watching for new ooms in manager Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.832458 2568 manager.go:349] Starting recovery of all containers Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.840708 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.840841 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-2.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.842448 2568 manager.go:354] Recovery completed Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.847387 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.849221 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.849240 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.849253 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.849898 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.849906 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.849919 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.850783 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.851911 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.851942 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.851965 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.852149 2568 policy_none.go:49] "None policy: Start" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.852163 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.852173 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.854176 2568 policy_none.go:47] "Start" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.862101 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-2.ec2.internal.18c4f34100501c0b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-2.ec2.internal,UID:ip-10-0-139-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-2.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-2.ec2.internal,},FirstTimestamp:2026-07-23 15:00:12.849232907 +0000 UTC m=+0.442683816,LastTimestamp:2026-07-23 15:00:12.849232907 +0000 UTC m=+0.442683816,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-2.ec2.internal,}" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.867428 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-2.ec2.internal.18c4f34100505864 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-2.ec2.internal,UID:ip-10-0-139-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-2.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-2.ec2.internal,},FirstTimestamp:2026-07-23 15:00:12.849248356 +0000 UTC m=+0.442699266,LastTimestamp:2026-07-23 15:00:12.849248356 +0000 UTC m=+0.442699266,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-2.ec2.internal,}" Jul 23 15:00:13.012028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.871914 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t67hw" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.877862 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-2.ec2.internal.18c4f34100507f5b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-2.ec2.internal,UID:ip-10-0-139-2.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-2.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-2.ec2.internal,},FirstTimestamp:2026-07-23 15:00:12.849258331 +0000 UTC m=+0.442709240,LastTimestamp:2026-07-23 15:00:12.849258331 +0000 UTC m=+0.442709240,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-2.ec2.internal,}" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.879511 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t67hw" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.909743 2568 manager.go:337] "Starting Device Plugin manager" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.909768 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.909783 2568 server.go:87] "Starting device plugin registration server" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.909940 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.909948 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.910749 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.910829 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.910838 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.911346 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.911383 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.956848 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.957964 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.958009 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.958027 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:12.958056 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:12.960726 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.011009 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.011749 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.011783 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.011801 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.011831 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.024134 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.024150 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-2.ec2.internal\": node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.056600 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.058839 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal"] Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.058891 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.059625 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.059649 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.059661 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062128 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062229 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062255 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062766 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062795 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062768 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.167337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062809 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062830 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.062843 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.065162 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.065192 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.065843 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.065866 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.065880 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.095815 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-2.ec2.internal\" not found" node="ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.100376 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-2.ec2.internal\" not found" node="ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.132346 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13d4d3160ab424e4c5acb044cbcb8ef0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal\" (UID: \"13d4d3160ab424e4c5acb044cbcb8ef0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.132368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13d4d3160ab424e4c5acb044cbcb8ef0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal\" (UID: \"13d4d3160ab424e4c5acb044cbcb8ef0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.132384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eeaa75121a38762b69786f5043319e69-config\") pod \"kube-apiserver-proxy-ip-10-0-139-2.ec2.internal\" (UID: \"eeaa75121a38762b69786f5043319e69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.168844 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.157474 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.331217 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.233273 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13d4d3160ab424e4c5acb044cbcb8ef0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal\" (UID: \"13d4d3160ab424e4c5acb044cbcb8ef0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.331217 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.233297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13d4d3160ab424e4c5acb044cbcb8ef0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal\" (UID: \"13d4d3160ab424e4c5acb044cbcb8ef0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.331217 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.233313 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eeaa75121a38762b69786f5043319e69-config\") pod \"kube-apiserver-proxy-ip-10-0-139-2.ec2.internal\" (UID: \"eeaa75121a38762b69786f5043319e69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.331217 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.233355 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eeaa75121a38762b69786f5043319e69-config\") pod \"kube-apiserver-proxy-ip-10-0-139-2.ec2.internal\" (UID: \"eeaa75121a38762b69786f5043319e69\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.331217 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.233378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13d4d3160ab424e4c5acb044cbcb8ef0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal\" (UID: \"13d4d3160ab424e4c5acb044cbcb8ef0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.331217 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.233397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13d4d3160ab424e4c5acb044cbcb8ef0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal\" (UID: \"13d4d3160ab424e4c5acb044cbcb8ef0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.331217 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.258373 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.485903 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.360313 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.485903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.400438 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.485903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.410896 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" Jul 23 15:00:13.485903 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.460584 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.627818 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.561720 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.775354 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.663206 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.775354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.735544 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 15:00:13.775354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.735683 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 15:00:13.775354 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.765069 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.830870 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.865509 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.868803 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.869671 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientMemory" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.869702 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.869713 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeHasSufficientPID" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.884447 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 14:55:13 +0000 UTC" deadline="2028-01-17 13:42:15.106682109 +0000 UTC" Jul 23 15:00:13.964428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:13.884475 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13030h42m1.222209796s" Jul 23 15:00:13.969971 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:13.969800 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:14.109017 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.070794 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:14.234719 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.171878 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:14.375655 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.272610 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-2.ec2.internal\" not found" Jul 23 15:00:14.375655 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.275335 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 15:00:14.375655 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.325308 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 15:00:14.499383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.421913 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" Jul 23 15:00:14.499383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.437598 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 15:00:14.499383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.438649 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" Jul 23 15:00:14.499383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.448950 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 15:00:14.629696 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:14.502385 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d4d3160ab424e4c5acb044cbcb8ef0.slice/crio-9727779e61fc264786b0ba0702f042b11e3c41b61abdecc2f2c3ba158fc81fc5 WatchSource:0}: Error finding container 9727779e61fc264786b0ba0702f042b11e3c41b61abdecc2f2c3ba158fc81fc5: Status 404 returned error can't find the container with id 9727779e61fc264786b0ba0702f042b11e3c41b61abdecc2f2c3ba158fc81fc5 Jul 23 15:00:14.629696 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:14.506156 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeaa75121a38762b69786f5043319e69.slice/crio-7ca33e79a06aacbea1865ee1b35573b9033bf71ead545681d08e9bcd367b8502 WatchSource:0}: Error finding container 7ca33e79a06aacbea1865ee1b35573b9033bf71ead545681d08e9bcd367b8502: Status 404 returned error can't find the container with id 7ca33e79a06aacbea1865ee1b35573b9033bf71ead545681d08e9bcd367b8502 Jul 23 15:00:14.629696 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.508204 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 15:00:14.794203 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.794183 2568 apiserver.go:52] "Watching apiserver" Jul 23 15:00:14.804469 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.804445 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 15:00:14.806706 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.806674 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-2p6st","openshift-ovn-kubernetes/ovnkube-node-66ftl","kube-system/global-pull-secret-syncer-g7spv","kube-system/konnectivity-agent-mprlv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl","openshift-dns/node-resolver-2tnf9","openshift-multus/multus-additional-cni-plugins-95l8l","openshift-multus/multus-rsrjp","openshift-multus/network-metrics-daemon-f6mzs","openshift-network-diagnostics/network-check-target-6cr64","kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dbnwn","openshift-image-registry/node-ca-vwksz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal"] Jul 23 15:00:14.810688 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.810671 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.813890 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.813863 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 15:00:14.814120 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.814094 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ndmjg\"" Jul 23 15:00:14.814260 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.814229 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 15:00:14.814335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.814229 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 15:00:14.814335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.814229 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 15:00:14.814557 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.814542 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 15:00:14.821752 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.821736 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.822875 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.822858 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 15:00:14.824111 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.824095 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.824203 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.824160 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:14.825359 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.825342 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 15:00:14.826090 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.826060 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 15:00:14.826256 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.826236 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 15:00:14.826365 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.826286 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z5gs5\"" Jul 23 15:00:14.826473 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.826458 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:14.828939 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.828919 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.829206 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.829151 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 15:00:14.829369 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.829353 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 15:00:14.834788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.830579 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5s4x4\"" Jul 23 15:00:14.834788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.831789 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lbhqk\"" Jul 23 15:00:14.834788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.832031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.834788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.832401 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 15:00:14.834788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.832648 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 15:00:14.834788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.834410 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:14.834788 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.834461 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.835783 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.835790 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.835828 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7phl2\"" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.836301 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.837283 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.839849 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.840971 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-sys-fs\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b36d684-b669-45b1-b436-4b2d52a6134a-tmp-dir\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841085 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-device-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrhsp\" (UniqueName: \"kubernetes.io/projected/90bad6ea-6d17-4d7c-8a28-675f399e05b5-kube-api-access-zrhsp\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841173 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svfz\" (UniqueName: \"kubernetes.io/projected/66136ccc-448f-4364-83e7-df4ba4703e61-kube-api-access-9svfz\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841182 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841202 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00c24181-cecc-4735-aa71-d217f7dcc3b1-kubelet-config\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e-agent-certs\") pod \"konnectivity-agent-mprlv\" (UID: \"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e\") " pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841268 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841286 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-registration-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-cnibin\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841332 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66136ccc-448f-4364-83e7-df4ba4703e61-host\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841345 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hsfvk\"" Jul 23 15:00:14.842321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e-konnectivity-ca\") pod \"konnectivity-agent-mprlv\" (UID: \"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e\") " pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b36d684-b669-45b1-b436-4b2d52a6134a-hosts-file\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxjf\" (UniqueName: \"kubernetes.io/projected/0b36d684-b669-45b1-b436-4b2d52a6134a-kube-api-access-2zxjf\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841490 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841489 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-os-release\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841535 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841566 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841624 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/2df20e09-01a7-488a-aaa8-1ce22e207283-kube-api-access-mqxc8\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841649 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66136ccc-448f-4364-83e7-df4ba4703e61-serviceca\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841714 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-system-cni-dir\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-cni-binary-copy\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00c24181-cecc-4735-aa71-d217f7dcc3b1-dbus\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.841769 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-socket-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.842710 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:14.843304 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.842763 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:14.843895 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.843439 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 15:00:14.843895 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.843601 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 15:00:14.845105 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.845037 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 15:00:14.845845 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.845422 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 15:00:14.845845 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.845513 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 15:00:14.846033 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.846012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.846103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.846059 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 15:00:14.846165 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.846108 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-smvq6\"" Jul 23 15:00:14.848039 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.847898 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 15:00:14.849115 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.849090 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.849493 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.849471 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bqxb6\"" Jul 23 15:00:14.849553 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.849475 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 15:00:14.856411 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.856393 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 15:00:14.856525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.856509 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nhw5p\"" Jul 23 15:00:14.856813 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.856798 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 15:00:14.880521 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.880481 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vqqc4" Jul 23 15:00:14.890033 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.890015 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vqqc4" Jul 23 15:00:14.932845 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.932825 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 15:00:14.942056 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-os-release\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.942125 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-var-lib-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.942125 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-conf-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.942125 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-daemon-config\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.942234 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/2df20e09-01a7-488a-aaa8-1ce22e207283-kube-api-access-mqxc8\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.942234 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942222 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-os-release\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.942315 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysconfig\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.942365 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942341 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-systemd\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.942409 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942376 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovn-node-metrics-cert\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.942451 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942408 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-cni-bin\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.942451 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-system-cni-dir\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.942556 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-cni-binary-copy\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.942556 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942510 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-system-cni-dir\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.942556 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00c24181-cecc-4735-aa71-d217f7dcc3b1-dbus\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.942712 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-socket-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.942712 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-sys-fs\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.942712 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b36d684-b669-45b1-b436-4b2d52a6134a-tmp-dir\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.942712 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-kubernetes\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.942712 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-device-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942712 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-sys-fs\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942728 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-socket-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovnkube-config\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942768 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00c24181-cecc-4735-aa71-d217f7dcc3b1-dbus\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-device-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-system-cni-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-cni-binary-copy\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-socket-dir-parent\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-netns\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.942954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-os-release\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942981 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-registration-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942979 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b36d684-b669-45b1-b436-4b2d52a6134a-tmp-dir\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.942995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec75d73-28ec-42c1-b427-51fa31c26e46-tmp\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943018 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-cni-binary-copy\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxwgm\" (UniqueName: \"kubernetes.io/projected/2ec75d73-28ec-42c1-b427-51fa31c26e46-kube-api-access-pxwgm\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943061 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-registration-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-k8s-cni-cncf-io\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-kubelet\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-etc-kubernetes\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66136ccc-448f-4364-83e7-df4ba4703e61-host\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e-konnectivity-ca\") pod \"konnectivity-agent-mprlv\" (UID: \"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e\") " pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943257 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66136ccc-448f-4364-83e7-df4ba4703e61-host\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943266 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-env-overrides\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-sys\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.943667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-etc-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-multus-certs\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943396 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943469 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-modprobe-d\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943520 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-log-socket\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-tuning-conf-dir\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66136ccc-448f-4364-83e7-df4ba4703e61-serviceca\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943632 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943681 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysctl-conf\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943715 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-run-netns\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943737 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-node-log\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.943690 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:14.944525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943752 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e-konnectivity-ca\") pod \"konnectivity-agent-mprlv\" (UID: \"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e\") " pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-cni-netd\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovnkube-script-lib\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:14.943814 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret podName:00c24181-cecc-4735-aa71-d217f7dcc3b1 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:15.443792425 +0000 UTC m=+3.037243340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret") pod "global-pull-secret-syncer-g7spv" (UID: "00c24181-cecc-4735-aa71-d217f7dcc3b1") : object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943840 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-kubelet\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-cni-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943881 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/90bad6ea-6d17-4d7c-8a28-675f399e05b5-etc-selinux\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943904 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-cni-multus\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943926 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2df20e09-01a7-488a-aaa8-1ce22e207283-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv6hm\" (UniqueName: \"kubernetes.io/projected/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-kube-api-access-tv6hm\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrhsp\" (UniqueName: \"kubernetes.io/projected/90bad6ea-6d17-4d7c-8a28-675f399e05b5-kube-api-access-zrhsp\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.943986 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6c6r\" (UniqueName: \"kubernetes.io/projected/daf7c9f6-b314-4849-b190-11ca4b83d438-kube-api-access-z6c6r\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944011 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944028 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysctl-d\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-run\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-lib-modules\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.945060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944096 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-host\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svfz\" (UniqueName: \"kubernetes.io/projected/66136ccc-448f-4364-83e7-df4ba4703e61-kube-api-access-9svfz\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944154 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-tuned\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944153 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/66136ccc-448f-4364-83e7-df4ba4703e61-serviceca\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tvv\" (UniqueName: \"kubernetes.io/projected/954165e8-54ba-4595-b41a-186d83bd2818-kube-api-access-b2tvv\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944185 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-systemd-units\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944207 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26wf2\" (UniqueName: \"kubernetes.io/projected/eecc2928-3d95-4b0b-8d39-2777137e21bf-kube-api-access-26wf2\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944230 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-hostroot\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00c24181-cecc-4735-aa71-d217f7dcc3b1-kubelet-config\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944267 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e-agent-certs\") pod \"konnectivity-agent-mprlv\" (UID: \"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e\") " pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daf7c9f6-b314-4849-b190-11ca4b83d438-host-slash\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944345 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00c24181-cecc-4735-aa71-d217f7dcc3b1-kubelet-config\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-systemd\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-ovn\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-cni-bin\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-cnibin\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:14.945564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-cnibin\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944556 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b36d684-b669-45b1-b436-4b2d52a6134a-hosts-file\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944562 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944600 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0b36d684-b669-45b1-b436-4b2d52a6134a-hosts-file\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944607 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2df20e09-01a7-488a-aaa8-1ce22e207283-cnibin\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944623 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxjf\" (UniqueName: \"kubernetes.io/projected/0b36d684-b669-45b1-b436-4b2d52a6134a-kube-api-access-2zxjf\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-slash\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/daf7c9f6-b314-4849-b190-11ca4b83d438-iptables-alerter-script\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:14.946040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.944678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-var-lib-kubelet\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:14.947603 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.947588 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e-agent-certs\") pod \"konnectivity-agent-mprlv\" (UID: \"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e\") " pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:14.956489 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.956467 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxc8\" (UniqueName: \"kubernetes.io/projected/2df20e09-01a7-488a-aaa8-1ce22e207283-kube-api-access-mqxc8\") pod \"multus-additional-cni-plugins-95l8l\" (UID: \"2df20e09-01a7-488a-aaa8-1ce22e207283\") " pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:14.956637 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.956605 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrhsp\" (UniqueName: \"kubernetes.io/projected/90bad6ea-6d17-4d7c-8a28-675f399e05b5-kube-api-access-zrhsp\") pod \"aws-ebs-csi-driver-node-lzfhl\" (UID: \"90bad6ea-6d17-4d7c-8a28-675f399e05b5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:14.960195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.960174 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxjf\" (UniqueName: \"kubernetes.io/projected/0b36d684-b669-45b1-b436-4b2d52a6134a-kube-api-access-2zxjf\") pod \"node-resolver-2tnf9\" (UID: \"0b36d684-b669-45b1-b436-4b2d52a6134a\") " pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:14.960569 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.960548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svfz\" (UniqueName: \"kubernetes.io/projected/66136ccc-448f-4364-83e7-df4ba4703e61-kube-api-access-9svfz\") pod \"node-ca-vwksz\" (UID: \"66136ccc-448f-4364-83e7-df4ba4703e61\") " pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:14.961990 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.961955 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" event={"ID":"eeaa75121a38762b69786f5043319e69","Type":"ContainerStarted","Data":"7ca33e79a06aacbea1865ee1b35573b9033bf71ead545681d08e9bcd367b8502"} Jul 23 15:00:14.963147 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:14.963126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" event={"ID":"13d4d3160ab424e4c5acb044cbcb8ef0","Type":"ContainerStarted","Data":"9727779e61fc264786b0ba0702f042b11e3c41b61abdecc2f2c3ba158fc81fc5"} Jul 23 15:00:15.045260 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-env-overrides\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-sys\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.045354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-etc-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045295 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-multus-certs\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-sys\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045362 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-etc-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-modprobe-d\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045406 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-log-socket\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045404 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-multus-certs\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045445 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-log-socket\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045449 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045496 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysctl-conf\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.045520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-run-netns\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045529 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-modprobe-d\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-run-ovn-kubernetes\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045558 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-node-log\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045578 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-run-netns\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045587 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-cni-netd\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045605 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-env-overrides\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovnkube-script-lib\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045620 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-node-log\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045619 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysctl-conf\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-cni-netd\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-kubelet\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045724 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-cni-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-cni-multus\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045788 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-kubelet\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-cni-multus\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv6hm\" (UniqueName: \"kubernetes.io/projected/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-kube-api-access-tv6hm\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.045823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045821 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-cni-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6c6r\" (UniqueName: \"kubernetes.io/projected/daf7c9f6-b314-4849-b190-11ca4b83d438-kube-api-access-z6c6r\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysctl-d\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-run\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-lib-modules\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-host\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.045972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-tuned\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046038 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-run\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046044 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-host\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovnkube-script-lib\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046075 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysctl-d\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-lib-modules\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046113 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tvv\" (UniqueName: \"kubernetes.io/projected/954165e8-54ba-4595-b41a-186d83bd2818-kube-api-access-b2tvv\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.046130 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046148 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-systemd-units\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.046629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26wf2\" (UniqueName: \"kubernetes.io/projected/eecc2928-3d95-4b0b-8d39-2777137e21bf-kube-api-access-26wf2\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.046186 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs podName:954165e8-54ba-4595-b41a-186d83bd2818 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:15.546173615 +0000 UTC m=+3.139624516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs") pod "network-metrics-daemon-f6mzs" (UID: "954165e8-54ba-4595-b41a-186d83bd2818") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-hostroot\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046220 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-systemd-units\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046229 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daf7c9f6-b314-4849-b190-11ca4b83d438-host-slash\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046256 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-hostroot\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-systemd\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daf7c9f6-b314-4849-b190-11ca4b83d438-host-slash\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-ovn\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-cni-bin\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-ovn\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046398 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-cnibin\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046409 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-systemd\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046416 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-cni-bin\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046426 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-slash\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/daf7c9f6-b314-4849-b190-11ca4b83d438-iptables-alerter-script\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-cnibin\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-var-lib-kubelet\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-var-lib-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046491 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-slash\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-conf-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046502 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-var-lib-kubelet\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046534 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-conf-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046540 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-var-lib-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046558 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-daemon-config\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysconfig\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-systemd\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046615 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovn-node-metrics-cert\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-cni-bin\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-kubernetes\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046660 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-systemd\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovnkube-config\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046696 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-system-cni-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-cni-binary-copy\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-socket-dir-parent\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.047775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-netns\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046842 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-os-release\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec75d73-28ec-42c1-b427-51fa31c26e46-tmp\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxwgm\" (UniqueName: \"kubernetes.io/projected/2ec75d73-28ec-42c1-b427-51fa31c26e46-kube-api-access-pxwgm\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-k8s-cni-cncf-io\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.046983 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-kubelet\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-etc-kubernetes\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047029 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovnkube-config\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047056 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/daf7c9f6-b314-4849-b190-11ca4b83d438-iptables-alerter-script\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047092 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-cni-bin\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047111 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-etc-kubernetes\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047116 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-sysconfig\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-kubernetes\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047165 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-system-cni-dir\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047221 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-daemon-config\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047230 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-os-release\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047298 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-k8s-cni-cncf-io\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047426 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-run-netns\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047471 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-cni-binary-copy\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047492 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-multus-socket-dir-parent\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047531 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-run-openvswitch\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.048925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047552 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-host-var-lib-kubelet\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.048925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.047628 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eecc2928-3d95-4b0b-8d39-2777137e21bf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.048925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.048396 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2ec75d73-28ec-42c1-b427-51fa31c26e46-etc-tuned\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.049230 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.049106 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2ec75d73-28ec-42c1-b427-51fa31c26e46-tmp\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.049230 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.049202 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eecc2928-3d95-4b0b-8d39-2777137e21bf-ovn-node-metrics-cert\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.054146 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.054130 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 15:00:15.054146 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.054148 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 15:00:15.054246 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.054158 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lxcws for pod openshift-network-diagnostics/network-check-target-6cr64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:15.054246 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.054207 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws podName:e4b67358-0326-47cb-bb64-03ba0411673d nodeName:}" failed. No retries permitted until 2026-07-23 15:00:15.554195515 +0000 UTC m=+3.147646439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lxcws" (UniqueName: "kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws") pod "network-check-target-6cr64" (UID: "e4b67358-0326-47cb-bb64-03ba0411673d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:15.056402 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.056378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv6hm\" (UniqueName: \"kubernetes.io/projected/f66b5f40-43fa-4d2e-a84f-da5a18d81b52-kube-api-access-tv6hm\") pod \"multus-rsrjp\" (UID: \"f66b5f40-43fa-4d2e-a84f-da5a18d81b52\") " pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.056500 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.056446 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6c6r\" (UniqueName: \"kubernetes.io/projected/daf7c9f6-b314-4849-b190-11ca4b83d438-kube-api-access-z6c6r\") pod \"iptables-alerter-2p6st\" (UID: \"daf7c9f6-b314-4849-b190-11ca4b83d438\") " pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:15.056705 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.056686 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tvv\" (UniqueName: \"kubernetes.io/projected/954165e8-54ba-4595-b41a-186d83bd2818-kube-api-access-b2tvv\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:15.057392 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.057370 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxwgm\" (UniqueName: \"kubernetes.io/projected/2ec75d73-28ec-42c1-b427-51fa31c26e46-kube-api-access-pxwgm\") pod \"tuned-dbnwn\" (UID: \"2ec75d73-28ec-42c1-b427-51fa31c26e46\") " pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.057857 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.057840 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26wf2\" (UniqueName: \"kubernetes.io/projected/eecc2928-3d95-4b0b-8d39-2777137e21bf-kube-api-access-26wf2\") pod \"ovnkube-node-66ftl\" (UID: \"eecc2928-3d95-4b0b-8d39-2777137e21bf\") " pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.120343 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.120315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-95l8l" Jul 23 15:00:15.129835 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.129817 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vwksz" Jul 23 15:00:15.130807 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.130748 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2df20e09_01a7_488a_aaa8_1ce22e207283.slice/crio-4b218baed0773f046a5f4cd1674174cc20734a2b2b7d9fb780a86713c4cf0aa2 WatchSource:0}: Error finding container 4b218baed0773f046a5f4cd1674174cc20734a2b2b7d9fb780a86713c4cf0aa2: Status 404 returned error can't find the container with id 4b218baed0773f046a5f4cd1674174cc20734a2b2b7d9fb780a86713c4cf0aa2 Jul 23 15:00:15.138027 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.138007 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66136ccc_448f_4364_83e7_df4ba4703e61.slice/crio-eddfba0344bfdab2a8aa9f537d60e232898ed8ebfe71738943b44ee0dd278a2a WatchSource:0}: Error finding container eddfba0344bfdab2a8aa9f537d60e232898ed8ebfe71738943b44ee0dd278a2a: Status 404 returned error can't find the container with id eddfba0344bfdab2a8aa9f537d60e232898ed8ebfe71738943b44ee0dd278a2a Jul 23 15:00:15.143220 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.143201 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:15.144322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.144305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" Jul 23 15:00:15.152429 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.152410 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2tnf9" Jul 23 15:00:15.152864 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.152843 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93bd9a2_2c0c_43d1_b80a_afcb17d9a49e.slice/crio-df71d8b986c4d0c05efdc15144f170f193dd2bd160ddd06f180bd9ad44f41755 WatchSource:0}: Error finding container df71d8b986c4d0c05efdc15144f170f193dd2bd160ddd06f180bd9ad44f41755: Status 404 returned error can't find the container with id df71d8b986c4d0c05efdc15144f170f193dd2bd160ddd06f180bd9ad44f41755 Jul 23 15:00:15.156820 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.156796 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bad6ea_6d17_4d7c_8a28_675f399e05b5.slice/crio-5cb51d145d2628775a799367aeb97c7a7ce4965a893071a1c8569cc2731adee9 WatchSource:0}: Error finding container 5cb51d145d2628775a799367aeb97c7a7ce4965a893071a1c8569cc2731adee9: Status 404 returned error can't find the container with id 5cb51d145d2628775a799367aeb97c7a7ce4965a893071a1c8569cc2731adee9 Jul 23 15:00:15.158593 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.158577 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2p6st" Jul 23 15:00:15.161714 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.161692 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b36d684_b669_45b1_b436_4b2d52a6134a.slice/crio-a12480afb480010c7d2a9133ec09c7dd1d377af12ff9682627e1a7adc55d8de0 WatchSource:0}: Error finding container a12480afb480010c7d2a9133ec09c7dd1d377af12ff9682627e1a7adc55d8de0: Status 404 returned error can't find the container with id a12480afb480010c7d2a9133ec09c7dd1d377af12ff9682627e1a7adc55d8de0 Jul 23 15:00:15.164044 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.164027 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:15.167788 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.167769 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf7c9f6_b314_4849_b190_11ca4b83d438.slice/crio-fd21dcb1ca533230bfbacc7b042c93119c178da5cab6dab54ea74c299101bff6 WatchSource:0}: Error finding container fd21dcb1ca533230bfbacc7b042c93119c178da5cab6dab54ea74c299101bff6: Status 404 returned error can't find the container with id fd21dcb1ca533230bfbacc7b042c93119c178da5cab6dab54ea74c299101bff6 Jul 23 15:00:15.169506 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.169487 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsrjp" Jul 23 15:00:15.172501 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.172483 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecc2928_3d95_4b0b_8d39_2777137e21bf.slice/crio-7cb2b92864e336304d7e857e9cce040f0460ad28abce24c9eed95b36873fcf3b WatchSource:0}: Error finding container 7cb2b92864e336304d7e857e9cce040f0460ad28abce24c9eed95b36873fcf3b: Status 404 returned error can't find the container with id 7cb2b92864e336304d7e857e9cce040f0460ad28abce24c9eed95b36873fcf3b Jul 23 15:00:15.174570 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.174545 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" Jul 23 15:00:15.179634 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.179612 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66b5f40_43fa_4d2e_a84f_da5a18d81b52.slice/crio-45c5b414ca3d4e537d2964c24bf60f9384bee73b18a147db33e7c7c8fa875ece WatchSource:0}: Error finding container 45c5b414ca3d4e537d2964c24bf60f9384bee73b18a147db33e7c7c8fa875ece: Status 404 returned error can't find the container with id 45c5b414ca3d4e537d2964c24bf60f9384bee73b18a147db33e7c7c8fa875ece Jul 23 15:00:15.183877 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:15.183847 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec75d73_28ec_42c1_b427_51fa31c26e46.slice/crio-86a75ff7360478d71a4b403428d901c9ed26dcf7b6fd674616193990d58c0206 WatchSource:0}: Error finding container 86a75ff7360478d71a4b403428d901c9ed26dcf7b6fd674616193990d58c0206: Status 404 returned error can't find the container with id 86a75ff7360478d71a4b403428d901c9ed26dcf7b6fd674616193990d58c0206 Jul 23 15:00:15.250883 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.250857 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 15:00:15.449313 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.449235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:15.449456 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.449412 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:15.449509 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.449497 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret podName:00c24181-cecc-4735-aa71-d217f7dcc3b1 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:16.449480072 +0000 UTC m=+4.042930975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret") pod "global-pull-secret-syncer-g7spv" (UID: "00c24181-cecc-4735-aa71-d217f7dcc3b1") : object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:15.550380 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.550348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:15.550542 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.550467 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:15.550542 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.550528 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs podName:954165e8-54ba-4595-b41a-186d83bd2818 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:16.550509512 +0000 UTC m=+4.143960415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs") pod "network-metrics-daemon-f6mzs" (UID: "954165e8-54ba-4595-b41a-186d83bd2818") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:15.651568 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.651540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:15.651739 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.651669 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 15:00:15.651739 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.651687 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 15:00:15.651739 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.651697 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lxcws for pod openshift-network-diagnostics/network-check-target-6cr64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:15.651881 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:15.651749 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws podName:e4b67358-0326-47cb-bb64-03ba0411673d nodeName:}" failed. No retries permitted until 2026-07-23 15:00:16.651730506 +0000 UTC m=+4.245181405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxcws" (UniqueName: "kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws") pod "network-check-target-6cr64" (UID: "e4b67358-0326-47cb-bb64-03ba0411673d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:15.893174 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.893070 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 14:55:15 +0000 UTC" deadline="2028-02-02 17:39:48.53174451 +0000 UTC" Jul 23 15:00:15.893174 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.893121 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13418h39m32.638629264s" Jul 23 15:00:15.967144 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.967099 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" event={"ID":"2ec75d73-28ec-42c1-b427-51fa31c26e46","Type":"ContainerStarted","Data":"86a75ff7360478d71a4b403428d901c9ed26dcf7b6fd674616193990d58c0206"} Jul 23 15:00:15.968571 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.968512 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsrjp" event={"ID":"f66b5f40-43fa-4d2e-a84f-da5a18d81b52","Type":"ContainerStarted","Data":"45c5b414ca3d4e537d2964c24bf60f9384bee73b18a147db33e7c7c8fa875ece"} Jul 23 15:00:15.974104 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.974057 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"7cb2b92864e336304d7e857e9cce040f0460ad28abce24c9eed95b36873fcf3b"} Jul 23 15:00:15.977827 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.977799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2p6st" event={"ID":"daf7c9f6-b314-4849-b190-11ca4b83d438","Type":"ContainerStarted","Data":"fd21dcb1ca533230bfbacc7b042c93119c178da5cab6dab54ea74c299101bff6"} Jul 23 15:00:15.980168 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.980142 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2tnf9" event={"ID":"0b36d684-b669-45b1-b436-4b2d52a6134a","Type":"ContainerStarted","Data":"a12480afb480010c7d2a9133ec09c7dd1d377af12ff9682627e1a7adc55d8de0"} Jul 23 15:00:15.982273 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.982249 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mprlv" event={"ID":"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e","Type":"ContainerStarted","Data":"df71d8b986c4d0c05efdc15144f170f193dd2bd160ddd06f180bd9ad44f41755"} Jul 23 15:00:15.983998 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.983971 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerStarted","Data":"4b218baed0773f046a5f4cd1674174cc20734a2b2b7d9fb780a86713c4cf0aa2"} Jul 23 15:00:15.985991 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.985967 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" event={"ID":"90bad6ea-6d17-4d7c-8a28-675f399e05b5","Type":"ContainerStarted","Data":"5cb51d145d2628775a799367aeb97c7a7ce4965a893071a1c8569cc2731adee9"} Jul 23 15:00:15.987534 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:15.987471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vwksz" event={"ID":"66136ccc-448f-4364-83e7-df4ba4703e61","Type":"ContainerStarted","Data":"eddfba0344bfdab2a8aa9f537d60e232898ed8ebfe71738943b44ee0dd278a2a"} Jul 23 15:00:16.458708 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.458669 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:16.458907 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.458889 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:16.458979 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.458960 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret podName:00c24181-cecc-4735-aa71-d217f7dcc3b1 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:18.458939688 +0000 UTC m=+6.052390606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret") pod "global-pull-secret-syncer-g7spv" (UID: "00c24181-cecc-4735-aa71-d217f7dcc3b1") : object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:16.559555 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.559471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:16.559704 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.559658 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:16.559769 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.559722 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs podName:954165e8-54ba-4595-b41a-186d83bd2818 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:18.559702279 +0000 UTC m=+6.153153186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs") pod "network-metrics-daemon-f6mzs" (UID: "954165e8-54ba-4595-b41a-186d83bd2818") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:16.660373 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.660339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:16.660557 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.660539 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 15:00:16.660649 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.660565 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 15:00:16.660649 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.660580 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lxcws for pod openshift-network-diagnostics/network-check-target-6cr64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:16.660649 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.660636 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws podName:e4b67358-0326-47cb-bb64-03ba0411673d nodeName:}" failed. No retries permitted until 2026-07-23 15:00:18.660618054 +0000 UTC m=+6.254068961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxcws" (UniqueName: "kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws") pod "network-check-target-6cr64" (UID: "e4b67358-0326-47cb-bb64-03ba0411673d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:16.894126 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.893884 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 14:55:15 +0000 UTC" deadline="2028-04-03 10:26:40.580312472 +0000 UTC" Jul 23 15:00:16.894508 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.894128 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14875h26m23.686190253s" Jul 23 15:00:16.960968 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.960941 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:16.961396 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.961377 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:16.961531 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.961508 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:16.961919 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:16.961901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:16.962016 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.961998 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:16.963256 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:16.962667 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:17.016568 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:17.016538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" event={"ID":"13d4d3160ab424e4c5acb044cbcb8ef0","Type":"ContainerStarted","Data":"2bda45a55001f89aa48cac21aa4dadda546d11c7abdb7e64b42ecfb159f01cf2"} Jul 23 15:00:18.476945 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:18.476903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:18.477382 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.477111 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:18.477382 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.477176 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret podName:00c24181-cecc-4735-aa71-d217f7dcc3b1 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:22.477156813 +0000 UTC m=+10.070607719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret") pod "global-pull-secret-syncer-g7spv" (UID: "00c24181-cecc-4735-aa71-d217f7dcc3b1") : object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:18.578060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:18.577523 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:18.578060 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.577672 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:18.578060 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.577729 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs podName:954165e8-54ba-4595-b41a-186d83bd2818 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:22.577710992 +0000 UTC m=+10.171161888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs") pod "network-metrics-daemon-f6mzs" (UID: "954165e8-54ba-4595-b41a-186d83bd2818") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:18.678788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:18.678710 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:18.678942 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.678890 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 15:00:18.678942 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.678910 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 15:00:18.678942 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.678924 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lxcws for pod openshift-network-diagnostics/network-check-target-6cr64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:18.679121 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.678985 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws podName:e4b67358-0326-47cb-bb64-03ba0411673d nodeName:}" failed. No retries permitted until 2026-07-23 15:00:22.678963769 +0000 UTC m=+10.272414714 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxcws" (UniqueName: "kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws") pod "network-check-target-6cr64" (UID: "e4b67358-0326-47cb-bb64-03ba0411673d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:18.961141 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:18.961068 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:18.961289 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.961216 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:18.961289 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:18.961234 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:18.961394 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.961367 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:18.961454 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:18.961437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:18.961545 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:18.961530 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:19.033193 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:19.033159 2568 generic.go:356] "Generic (PLEG): container finished" podID="13d4d3160ab424e4c5acb044cbcb8ef0" containerID="2bda45a55001f89aa48cac21aa4dadda546d11c7abdb7e64b42ecfb159f01cf2" exitCode=0 Jul 23 15:00:19.033358 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:19.033205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" event={"ID":"13d4d3160ab424e4c5acb044cbcb8ef0","Type":"ContainerDied","Data":"2bda45a55001f89aa48cac21aa4dadda546d11c7abdb7e64b42ecfb159f01cf2"} Jul 23 15:00:20.959046 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:20.958881 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:20.959046 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:20.959035 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:20.959547 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:20.959351 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:20.959547 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:20.959472 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:20.959547 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:20.959512 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:20.959701 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:20.959590 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:22.511819 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:22.511784 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:22.512336 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.511938 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:22.512336 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.512017 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret podName:00c24181-cecc-4735-aa71-d217f7dcc3b1 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:30.511982718 +0000 UTC m=+18.105433631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret") pod "global-pull-secret-syncer-g7spv" (UID: "00c24181-cecc-4735-aa71-d217f7dcc3b1") : object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:22.613092 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:22.613063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:22.613260 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.613241 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:22.613332 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.613320 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs podName:954165e8-54ba-4595-b41a-186d83bd2818 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:30.613300328 +0000 UTC m=+18.206751231 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs") pod "network-metrics-daemon-f6mzs" (UID: "954165e8-54ba-4595-b41a-186d83bd2818") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:22.713606 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:22.713576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:22.713821 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.713768 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 15:00:22.713821 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.713795 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 15:00:22.713821 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.713810 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lxcws for pod openshift-network-diagnostics/network-check-target-6cr64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:22.713974 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.713865 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws podName:e4b67358-0326-47cb-bb64-03ba0411673d nodeName:}" failed. No retries permitted until 2026-07-23 15:00:30.713848421 +0000 UTC m=+18.307299319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxcws" (UniqueName: "kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws") pod "network-check-target-6cr64" (UID: "e4b67358-0326-47cb-bb64-03ba0411673d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:22.959542 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:22.959478 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:22.963625 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.960039 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:22.963625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:22.960404 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:22.963625 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.960555 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:22.963625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:22.961062 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:22.963625 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:22.961426 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:24.958914 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:24.958882 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:24.959454 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:24.958883 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:24.959454 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:24.959020 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:24.959454 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:24.959041 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:24.959454 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:24.959163 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:24.959454 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:24.959241 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:26.958598 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:26.958568 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:26.958598 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:26.958588 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:26.959046 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:26.958576 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:26.959046 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:26.958666 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:26.959046 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:26.958785 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:26.959046 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:26.958867 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:28.958439 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:28.958407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:28.958439 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:28.958441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:28.958938 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:28.958407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:28.958938 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:28.958531 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:28.958938 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:28.958617 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:28.958938 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:28.958683 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:30.575529 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:30.575031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:30.575529 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.575210 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:30.575529 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.575266 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret podName:00c24181-cecc-4735-aa71-d217f7dcc3b1 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:46.575249726 +0000 UTC m=+34.168700637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret") pod "global-pull-secret-syncer-g7spv" (UID: "00c24181-cecc-4735-aa71-d217f7dcc3b1") : object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:30.676293 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:30.676270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:30.676394 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.676361 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:30.676439 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.676404 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs podName:954165e8-54ba-4595-b41a-186d83bd2818 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:46.676391879 +0000 UTC m=+34.269842777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs") pod "network-metrics-daemon-f6mzs" (UID: "954165e8-54ba-4595-b41a-186d83bd2818") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:30.777005 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:30.776966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:30.777112 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.777095 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 15:00:30.777157 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.777116 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 15:00:30.777157 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.777128 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lxcws for pod openshift-network-diagnostics/network-check-target-6cr64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:30.777217 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.777169 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws podName:e4b67358-0326-47cb-bb64-03ba0411673d nodeName:}" failed. No retries permitted until 2026-07-23 15:00:46.77715775 +0000 UTC m=+34.370608652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxcws" (UniqueName: "kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws") pod "network-check-target-6cr64" (UID: "e4b67358-0326-47cb-bb64-03ba0411673d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:30.958978 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:30.958953 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:30.959067 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:30.958976 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:30.959067 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:30.958964 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:30.959149 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.959076 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:30.959204 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.959141 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:30.959251 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:30.959231 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:32.959377 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:32.959341 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:32.959846 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:32.959455 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:32.959846 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:32.959465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:32.959846 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:32.959496 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:32.959846 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:32.959586 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:32.959846 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:32.959678 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:34.958987 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:34.958955 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:34.959453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:34.958956 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:34.959453 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:34.959084 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:34.959453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:34.959104 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:34.959453 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:34.959185 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:34.959453 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:34.959286 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:36.067962 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.067507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" event={"ID":"2ec75d73-28ec-42c1-b427-51fa31c26e46","Type":"ContainerStarted","Data":"e65d30c3850732156876208dd50edde6d659d882b5a5744fba0fd09cc1d1a5c0"} Jul 23 15:00:36.069455 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.069427 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsrjp" event={"ID":"f66b5f40-43fa-4d2e-a84f-da5a18d81b52","Type":"ContainerStarted","Data":"7042add06ed130205bcdfa6b8feee29dbe916a81a2cf58a35ee1a4146dddde6c"} Jul 23 15:00:36.072897 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.072876 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:00:36.073344 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.073289 2568 generic.go:356] "Generic (PLEG): container finished" podID="eecc2928-3d95-4b0b-8d39-2777137e21bf" containerID="0a68a7a33b23983416faedcd2f0862b83ed7c786e085a49328ab6931dcc5b269" exitCode=1 Jul 23 15:00:36.073462 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.073370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"d6f361563ddf344d32397deee9b2ed8290def1accaff11059573daad09c9f151"} Jul 23 15:00:36.073462 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.073451 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"e3c504e22a5eee0c86c1bea442eb0fce84427999f41bbb15579edcd2cfce65f5"} Jul 23 15:00:36.073573 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.073498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"2950bab4a40d7a05d7269b64cf37d0845f873cfcd7c2d398790cf498a0262c3a"} Jul 23 15:00:36.073573 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.073545 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"70386d1b5eedab1c6cf7612e474ca93de81b456283aa92970c515e5802dbaada"} Jul 23 15:00:36.073672 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.073590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerDied","Data":"0a68a7a33b23983416faedcd2f0862b83ed7c786e085a49328ab6931dcc5b269"} Jul 23 15:00:36.073672 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.073640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"c6b3dcf3adc96197cd9713db039edd97bdf15f50ea93882c037959c628b62b88"} Jul 23 15:00:36.075097 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.075078 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2tnf9" event={"ID":"0b36d684-b669-45b1-b436-4b2d52a6134a","Type":"ContainerStarted","Data":"0d541bf442835d7de3fb482dc2346f167dba005f58aca8a9bdc69c2dba8e7d81"} Jul 23 15:00:36.076527 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.076507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mprlv" event={"ID":"a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e","Type":"ContainerStarted","Data":"c72e1d5287c60463a9c5b15a80927702725f422d78ab8152639a7ee2a7a17d72"} Jul 23 15:00:36.077976 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.077954 2568 generic.go:356] "Generic (PLEG): container finished" podID="2df20e09-01a7-488a-aaa8-1ce22e207283" containerID="c839c12b275ca766dcce8926cd52fe497d868894a814fa453b9b2a8c9f9790ae" exitCode=0 Jul 23 15:00:36.078085 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.077984 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerDied","Data":"c839c12b275ca766dcce8926cd52fe497d868894a814fa453b9b2a8c9f9790ae"} Jul 23 15:00:36.079969 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.079944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" event={"ID":"13d4d3160ab424e4c5acb044cbcb8ef0","Type":"ContainerStarted","Data":"6b69946298ddf4bf40ba0b25dd0db7040f43fd82f912f4b3e1b484edc068b24b"} Jul 23 15:00:36.081581 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.081556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" event={"ID":"90bad6ea-6d17-4d7c-8a28-675f399e05b5","Type":"ContainerStarted","Data":"3c7d8e1b5d3105c0e148f91dccb568607e930b16638fbc29ca18194b977f3d21"} Jul 23 15:00:36.083198 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.083172 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vwksz" event={"ID":"66136ccc-448f-4364-83e7-df4ba4703e61","Type":"ContainerStarted","Data":"b6bef29d484f995a0f53741bff42d2cbc027b28b86762c76cfe8200678101045"} Jul 23 15:00:36.084804 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.084781 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" event={"ID":"eeaa75121a38762b69786f5043319e69","Type":"ContainerStarted","Data":"251ef744d27db99830786d2ae8126488a6f43c8b51675578d4732d376b6eb849"} Jul 23 15:00:36.088119 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.088074 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dbnwn" podStartSLOduration=3.058975236 podStartE2EDuration="23.088057769s" totalImagesPullingTime="20.029082533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:36.087436699 +0000 UTC m=+23.680887619" watchObservedRunningTime="2026-07-23 15:00:36.088057769 +0000 UTC m=+23.681508688" Jul 23 15:00:36.104022 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.103968 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-2.ec2.internal" podStartSLOduration=22.103956671 podStartE2EDuration="22.103956671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:36.103841579 +0000 UTC m=+23.697292499" watchObservedRunningTime="2026-07-23 15:00:36.103956671 +0000 UTC m=+23.697407589" Jul 23 15:00:36.121518 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.121480 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-2.ec2.internal" podStartSLOduration=22.121469276 podStartE2EDuration="22.121469276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:36.121405511 +0000 UTC m=+23.714856430" watchObservedRunningTime="2026-07-23 15:00:36.121469276 +0000 UTC m=+23.714920195" Jul 23 15:00:36.167971 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.167934 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vwksz" podStartSLOduration=3.126617004 podStartE2EDuration="23.167921799s" totalImagesPullingTime="20.041304795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:36.167313362 +0000 UTC m=+23.760764280" watchObservedRunningTime="2026-07-23 15:00:36.167921799 +0000 UTC m=+23.761372718" Jul 23 15:00:36.185940 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.185905 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2tnf9" podStartSLOduration=3.123507052 podStartE2EDuration="23.185893907s" totalImagesPullingTime="20.062386855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:36.185850765 +0000 UTC m=+23.779301683" watchObservedRunningTime="2026-07-23 15:00:36.185893907 +0000 UTC m=+23.779344825" Jul 23 15:00:36.225736 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.225704 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mprlv" podStartSLOduration=11.791421795 podStartE2EDuration="23.225692497s" totalImagesPullingTime="11.434270702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:36.225429017 +0000 UTC m=+23.818879936" watchObservedRunningTime="2026-07-23 15:00:36.225692497 +0000 UTC m=+23.819143413" Jul 23 15:00:36.225945 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.225921 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rsrjp" podStartSLOduration=2.966878994 podStartE2EDuration="23.225917515s" totalImagesPullingTime="20.259038521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:36.206695396 +0000 UTC m=+23.800146316" watchObservedRunningTime="2026-07-23 15:00:36.225917515 +0000 UTC m=+23.819368461" Jul 23 15:00:36.763325 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.763304 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 15:00:36.926618 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.926517 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T15:00:36.763320953Z","UUID":"f7a5a75a-668f-4ffe-8c8c-d85ee465c4fd","Handler":null,"Name":"","Endpoint":""} Jul 23 15:00:36.928316 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.928283 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 15:00:36.928434 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.928328 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 15:00:36.958679 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.958649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:36.958788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.958654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:36.958788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:36.958767 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:36.958881 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:36.958739 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:36.958881 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:36.958858 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:36.958978 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:36.958946 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:37.088966 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:37.088897 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2p6st" event={"ID":"daf7c9f6-b314-4849-b190-11ca4b83d438","Type":"ContainerStarted","Data":"449394e8dafb944da0728d759596ea287f4bc8b20913e6fe5382530b7620d53c"} Jul 23 15:00:37.090864 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:37.090840 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" event={"ID":"90bad6ea-6d17-4d7c-8a28-675f399e05b5","Type":"ContainerStarted","Data":"2f3b512927a456b794f5845cf8ef949fa2c5eda1fba55852f8cc909805af5797"} Jul 23 15:00:37.107846 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:37.107811 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2p6st" podStartSLOduration=4.062442716 podStartE2EDuration="24.107799863s" totalImagesPullingTime="20.045357147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:37.107401897 +0000 UTC m=+24.700852816" watchObservedRunningTime="2026-07-23 15:00:37.107799863 +0000 UTC m=+24.701250783" Jul 23 15:00:38.165892 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:38.165696 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:38.166387 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:38.166366 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:38.958339 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:38.958312 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:38.958339 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:38.958309 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:38.958667 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:38.958316 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:38.958667 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:38.958521 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:38.958667 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:38.958541 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:38.958667 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:38.958601 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:39.098281 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:39.098255 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:00:39.098689 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:39.098656 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"47fc1c85b7773cbad120e54b8b3790c9d5634b0c2bcae0e4ceeb552550c46404"} Jul 23 15:00:39.100958 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:39.100930 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" event={"ID":"90bad6ea-6d17-4d7c-8a28-675f399e05b5","Type":"ContainerStarted","Data":"a99de446a8f01ffd9ead59e584a9c0931efe5dec317a3b82c1b0395c7167b1fd"} Jul 23 15:00:39.101133 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:39.101111 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:39.101574 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:39.101554 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mprlv" Jul 23 15:00:39.121847 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:39.121805 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lzfhl" podStartSLOduration=3.283509067 podStartE2EDuration="26.121789389s" totalImagesPullingTime="22.838280322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:39.1217194 +0000 UTC m=+26.715170322" watchObservedRunningTime="2026-07-23 15:00:39.121789389 +0000 UTC m=+26.715240309" Jul 23 15:00:40.959188 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:40.958965 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:40.959816 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:40.958973 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:40.959816 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:40.959337 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:40.959816 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:40.958987 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:40.959816 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:40.959219 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:40.959816 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:40.959437 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:41.106719 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:41.106691 2568 generic.go:356] "Generic (PLEG): container finished" podID="2df20e09-01a7-488a-aaa8-1ce22e207283" containerID="d4ce58277330e6b25256006932c5e15c1a39495f65867c7202da5ce59e89b09f" exitCode=0 Jul 23 15:00:41.106891 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:41.106764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerDied","Data":"d4ce58277330e6b25256006932c5e15c1a39495f65867c7202da5ce59e89b09f"} Jul 23 15:00:41.110224 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:41.110207 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:00:41.110638 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:41.110612 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"078e9a6d7f2bf17de722584c7608298dadecc7a311f1761a5584a3b47bbea2a6"} Jul 23 15:00:41.110778 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:41.110759 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:41.111006 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:41.110980 2568 scope.go:117] "RemoveContainer" containerID="0a68a7a33b23983416faedcd2f0862b83ed7c786e085a49328ab6931dcc5b269" Jul 23 15:00:42.115316 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.115292 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:00:42.115805 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.115647 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" event={"ID":"eecc2928-3d95-4b0b-8d39-2777137e21bf","Type":"ContainerStarted","Data":"f4ae1d9a483dc9a774de2066ebe15d8923b2a021aab7463231c30eac9a3ec4b2"} Jul 23 15:00:42.115906 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.115888 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:42.115962 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.115951 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:42.130350 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.130327 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:42.130477 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.130455 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:00:42.151338 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.151293 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" podStartSLOduration=8.955945823 podStartE2EDuration="29.151275582s" totalImagesPullingTime="20.195329759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:42.148621443 +0000 UTC m=+29.742072361" watchObservedRunningTime="2026-07-23 15:00:42.151275582 +0000 UTC m=+29.744726500" Jul 23 15:00:42.717022 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.716980 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7spv"] Jul 23 15:00:42.717208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.717101 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:42.717208 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:42.717179 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:42.720280 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.720259 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6cr64"] Jul 23 15:00:42.720377 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.720339 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:42.720414 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:42.720405 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:42.722966 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.722940 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6mzs"] Jul 23 15:00:42.723081 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:42.723065 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:42.723182 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:42.723155 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:43.119672 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:43.119594 2568 generic.go:356] "Generic (PLEG): container finished" podID="2df20e09-01a7-488a-aaa8-1ce22e207283" containerID="263dc280eb8561a34d274d86581ba332a0be1dad5e166d6f39d8523c82091a15" exitCode=0 Jul 23 15:00:43.120039 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:43.119673 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerDied","Data":"263dc280eb8561a34d274d86581ba332a0be1dad5e166d6f39d8523c82091a15"} Jul 23 15:00:43.958355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:43.958324 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:43.958355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:43.958348 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:43.958571 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:43.958414 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:43.958571 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:43.958554 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:44.124437 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:44.124395 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerStarted","Data":"e231d0527984c7d2f34cb54bc625dc7e34c094920b4a5e079a4574bad288dc00"} Jul 23 15:00:44.959270 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:44.959242 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:44.959428 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:44.959347 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:45.128769 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:45.128736 2568 generic.go:356] "Generic (PLEG): container finished" podID="2df20e09-01a7-488a-aaa8-1ce22e207283" containerID="e231d0527984c7d2f34cb54bc625dc7e34c094920b4a5e079a4574bad288dc00" exitCode=0 Jul 23 15:00:45.129426 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:45.128777 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerDied","Data":"e231d0527984c7d2f34cb54bc625dc7e34c094920b4a5e079a4574bad288dc00"} Jul 23 15:00:45.958887 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:45.958849 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:45.959091 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:45.958880 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:45.959091 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:45.958972 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:45.959091 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:45.959059 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:46.595205 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:46.595168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:46.595704 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.595338 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:46.595704 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.595419 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret podName:00c24181-cecc-4735-aa71-d217f7dcc3b1 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:18.595397822 +0000 UTC m=+66.188848736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret") pod "global-pull-secret-syncer-g7spv" (UID: "00c24181-cecc-4735-aa71-d217f7dcc3b1") : object "kube-system"/"original-pull-secret" not registered Jul 23 15:00:46.695984 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:46.695951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:46.696162 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.696091 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:46.696208 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.696162 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs podName:954165e8-54ba-4595-b41a-186d83bd2818 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:18.696140984 +0000 UTC m=+66.289591896 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs") pod "network-metrics-daemon-f6mzs" (UID: "954165e8-54ba-4595-b41a-186d83bd2818") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 15:00:46.796852 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:46.796805 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:46.797083 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.796947 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 15:00:46.797083 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.796967 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 15:00:46.797083 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.796977 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lxcws for pod openshift-network-diagnostics/network-check-target-6cr64: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:46.797083 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.797039 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws podName:e4b67358-0326-47cb-bb64-03ba0411673d nodeName:}" failed. No retries permitted until 2026-07-23 15:01:18.797023423 +0000 UTC m=+66.390474320 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lxcws" (UniqueName: "kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws") pod "network-check-target-6cr64" (UID: "e4b67358-0326-47cb-bb64-03ba0411673d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 15:00:46.959202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:46.959169 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:46.959378 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:46.959303 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f6mzs" podUID="954165e8-54ba-4595-b41a-186d83bd2818" Jul 23 15:00:47.958438 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:47.958411 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:47.958873 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:47.958415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:47.958873 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:47.958513 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6cr64" podUID="e4b67358-0326-47cb-bb64-03ba0411673d" Jul 23 15:00:47.958873 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:47.958633 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7spv" podUID="00c24181-cecc-4735-aa71-d217f7dcc3b1" Jul 23 15:00:48.214351 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.214323 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-2.ec2.internal" event="NodeReady" Jul 23 15:00:48.214513 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.214472 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 15:00:48.255048 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.255022 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bcd859484-9xr7k"] Jul 23 15:00:48.258145 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.258122 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.260728 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.260705 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 23 15:00:48.260869 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.260744 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 23 15:00:48.260869 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.260713 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 23 15:00:48.261268 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.261249 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qpm5s\"" Jul 23 15:00:48.268139 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.268117 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 23 15:00:48.273093 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.273064 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bcd859484-9xr7k"] Jul 23 15:00:48.279036 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.279015 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8792c"] Jul 23 15:00:48.281084 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.281064 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.283586 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.283560 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 15:00:48.283695 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.283678 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4gdwl\"" Jul 23 15:00:48.283792 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.283777 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 15:00:48.283870 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.283683 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 15:00:48.295236 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.293397 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8792c"] Jul 23 15:00:48.378738 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.378706 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tq87x"] Jul 23 15:00:48.381160 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.381141 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.383823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.383776 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 15:00:48.383954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.383929 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-msfpg\"" Jul 23 15:00:48.383954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.383942 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 15:00:48.393658 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.393612 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tq87x"] Jul 23 15:00:48.411503 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411477 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-certificates\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.411592 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-image-registry-private-configuration\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.411592 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411536 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.411592 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411566 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7a2dd24-8428-4072-8938-eeeb2895ecc5-ca-trust-extracted\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.411592 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-installation-pull-secrets\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.411811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411620 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af26b3d-e245-494b-b21b-5cb771291790-cert\") pod \"ingress-canary-8792c\" (UID: \"1af26b3d-e245-494b-b21b-5cb771291790\") " pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.411811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411643 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-trusted-ca\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.411811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9n5\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-kube-api-access-qs9n5\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.411811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411751 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-852qj\" (UniqueName: \"kubernetes.io/projected/1af26b3d-e245-494b-b21b-5cb771291790-kube-api-access-852qj\") pod \"ingress-canary-8792c\" (UID: \"1af26b3d-e245-494b-b21b-5cb771291790\") " pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.411811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.411794 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-bound-sa-token\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.512955 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.512874 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.512955 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.512936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7a2dd24-8428-4072-8938-eeeb2895ecc5-ca-trust-extracted\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513180 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513055 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-installation-pull-secrets\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513180 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af26b3d-e245-494b-b21b-5cb771291790-cert\") pod \"ingress-canary-8792c\" (UID: \"1af26b3d-e245-494b-b21b-5cb771291790\") " pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.513180 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-trusted-ca\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513180 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513159 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9n5\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-kube-api-access-qs9n5\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3024aa-1fca-4b32-825a-ac2b80fa4251-tmp-dir\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.513355 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:48.513067 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 15:00:48.513355 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:48.513209 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcd859484-9xr7k: secret "image-registry-tls" not found Jul 23 15:00:48.513355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513220 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-852qj\" (UniqueName: \"kubernetes.io/projected/1af26b3d-e245-494b-b21b-5cb771291790-kube-api-access-852qj\") pod \"ingress-canary-8792c\" (UID: \"1af26b3d-e245-494b-b21b-5cb771291790\") " pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.513355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-bound-sa-token\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513355 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:48.513290 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls podName:a7a2dd24-8428-4072-8938-eeeb2895ecc5 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:49.013267312 +0000 UTC m=+36.606718236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls") pod "image-registry-7bcd859484-9xr7k" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5") : secret "image-registry-tls" not found Jul 23 15:00:48.513635 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7a2dd24-8428-4072-8938-eeeb2895ecc5-ca-trust-extracted\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513635 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hj8\" (UniqueName: \"kubernetes.io/projected/0a3024aa-1fca-4b32-825a-ac2b80fa4251-kube-api-access-b9hj8\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.513635 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-certificates\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513635 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-image-registry-private-configuration\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.513635 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.513864 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.513641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3024aa-1fca-4b32-825a-ac2b80fa4251-config-volume\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.514158 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.514132 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-certificates\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.514285 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.514263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-trusted-ca\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.518035 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.517846 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-image-registry-private-configuration\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.518134 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.517861 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-installation-pull-secrets\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.518134 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.517893 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1af26b3d-e245-494b-b21b-5cb771291790-cert\") pod \"ingress-canary-8792c\" (UID: \"1af26b3d-e245-494b-b21b-5cb771291790\") " pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.522641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.522601 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-bound-sa-token\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.523272 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.523227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9n5\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-kube-api-access-qs9n5\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:48.523410 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.523362 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-852qj\" (UniqueName: \"kubernetes.io/projected/1af26b3d-e245-494b-b21b-5cb771291790-kube-api-access-852qj\") pod \"ingress-canary-8792c\" (UID: \"1af26b3d-e245-494b-b21b-5cb771291790\") " pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.590354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.590315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8792c" Jul 23 15:00:48.614831 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.614797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3024aa-1fca-4b32-825a-ac2b80fa4251-tmp-dir\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.614970 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.614863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hj8\" (UniqueName: \"kubernetes.io/projected/0a3024aa-1fca-4b32-825a-ac2b80fa4251-kube-api-access-b9hj8\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.614970 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.614902 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.614970 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.614927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3024aa-1fca-4b32-825a-ac2b80fa4251-config-volume\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.615145 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:48.615039 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 15:00:48.615145 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:48.615119 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls podName:0a3024aa-1fca-4b32-825a-ac2b80fa4251 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:49.115097457 +0000 UTC m=+36.708548365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls") pod "dns-default-tq87x" (UID: "0a3024aa-1fca-4b32-825a-ac2b80fa4251") : secret "dns-default-metrics-tls" not found Jul 23 15:00:48.615145 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.615124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3024aa-1fca-4b32-825a-ac2b80fa4251-tmp-dir\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.615465 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.615442 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0a3024aa-1fca-4b32-825a-ac2b80fa4251-config-volume\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.624130 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.624107 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hj8\" (UniqueName: \"kubernetes.io/projected/0a3024aa-1fca-4b32-825a-ac2b80fa4251-kube-api-access-b9hj8\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:48.958341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.958308 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:00:48.962022 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.961976 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 15:00:48.962416 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:48.961976 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qbp9q\"" Jul 23 15:00:49.018209 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.018178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:49.018337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:49.018315 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 15:00:49.018337 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:49.018330 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcd859484-9xr7k: secret "image-registry-tls" not found Jul 23 15:00:49.018412 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:49.018391 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls podName:a7a2dd24-8428-4072-8938-eeeb2895ecc5 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:50.01837343 +0000 UTC m=+37.611824333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls") pod "image-registry-7bcd859484-9xr7k" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5") : secret "image-registry-tls" not found Jul 23 15:00:49.119172 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.119136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:49.119333 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:49.119322 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 15:00:49.119384 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:49.119377 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls podName:0a3024aa-1fca-4b32-825a-ac2b80fa4251 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:50.119363094 +0000 UTC m=+37.712813994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls") pod "dns-default-tq87x" (UID: "0a3024aa-1fca-4b32-825a-ac2b80fa4251") : secret "dns-default-metrics-tls" not found Jul 23 15:00:49.493073 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.493041 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8792c"] Jul 23 15:00:49.497316 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:49.497287 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af26b3d_e245_494b_b21b_5cb771291790.slice/crio-a2257938caaec3de9219137d5589ac9a58fbfd95f43c105044ef062031c6f315 WatchSource:0}: Error finding container a2257938caaec3de9219137d5589ac9a58fbfd95f43c105044ef062031c6f315: Status 404 returned error can't find the container with id a2257938caaec3de9219137d5589ac9a58fbfd95f43c105044ef062031c6f315 Jul 23 15:00:49.959105 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.959079 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:00:49.959247 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.959079 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:00:49.962484 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.962456 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 15:00:49.963624 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.963607 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 15:00:49.963624 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.963613 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 15:00:49.963777 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:49.963653 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v5f82\"" Jul 23 15:00:50.026676 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:50.026644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:50.026813 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:50.026782 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 15:00:50.026813 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:50.026797 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcd859484-9xr7k: secret "image-registry-tls" not found Jul 23 15:00:50.026891 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:50.026846 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls podName:a7a2dd24-8428-4072-8938-eeeb2895ecc5 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:52.026831912 +0000 UTC m=+39.620282812 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls") pod "image-registry-7bcd859484-9xr7k" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5") : secret "image-registry-tls" not found Jul 23 15:00:50.127930 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:50.127904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:50.128075 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:50.128048 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 15:00:50.128117 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:50.128097 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls podName:0a3024aa-1fca-4b32-825a-ac2b80fa4251 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:52.12808562 +0000 UTC m=+39.721536516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls") pod "dns-default-tq87x" (UID: "0a3024aa-1fca-4b32-825a-ac2b80fa4251") : secret "dns-default-metrics-tls" not found Jul 23 15:00:50.142336 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:50.142310 2568 generic.go:356] "Generic (PLEG): container finished" podID="2df20e09-01a7-488a-aaa8-1ce22e207283" containerID="cfadd98275eb654ee53c2ce6b1474d4c23535c3c29b1bf6460dede9f794be7b4" exitCode=0 Jul 23 15:00:50.142435 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:50.142394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerDied","Data":"cfadd98275eb654ee53c2ce6b1474d4c23535c3c29b1bf6460dede9f794be7b4"} Jul 23 15:00:50.143501 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:50.143482 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8792c" event={"ID":"1af26b3d-e245-494b-b21b-5cb771291790","Type":"ContainerStarted","Data":"a2257938caaec3de9219137d5589ac9a58fbfd95f43c105044ef062031c6f315"} Jul 23 15:00:51.149496 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:51.149455 2568 generic.go:356] "Generic (PLEG): container finished" podID="2df20e09-01a7-488a-aaa8-1ce22e207283" containerID="90003c6d2bf7d2ed19abb4920eca5dd8b6b13b02be983d795444d563dc86e89b" exitCode=0 Jul 23 15:00:51.149955 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:51.149510 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerDied","Data":"90003c6d2bf7d2ed19abb4920eca5dd8b6b13b02be983d795444d563dc86e89b"} Jul 23 15:00:52.041616 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.041418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:52.041797 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.041566 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 15:00:52.041797 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.041649 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcd859484-9xr7k: secret "image-registry-tls" not found Jul 23 15:00:52.041797 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.041706 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls podName:a7a2dd24-8428-4072-8938-eeeb2895ecc5 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:56.041685238 +0000 UTC m=+43.635136155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls") pod "image-registry-7bcd859484-9xr7k" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5") : secret "image-registry-tls" not found Jul 23 15:00:52.095883 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.095855 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l"] Jul 23 15:00:52.099907 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.099884 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb"] Jul 23 15:00:52.100052 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.100032 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.102489 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.102462 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 23 15:00:52.102608 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.102528 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 23 15:00:52.102882 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.102858 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:52.102986 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.102925 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fgvt8\"" Jul 23 15:00:52.103079 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.103036 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 23 15:00:52.103428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.103410 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 23 15:00:52.105203 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.105187 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6fqx9\"" Jul 23 15:00:52.105400 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.105372 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 23 15:00:52.105614 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.105598 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 23 15:00:52.106425 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.106408 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 23 15:00:52.111593 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.111554 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l"] Jul 23 15:00:52.112468 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.112448 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb"] Jul 23 15:00:52.142955 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.142922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:52.143068 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.143056 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 15:00:52.143134 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.143118 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls podName:0a3024aa-1fca-4b32-825a-ac2b80fa4251 nodeName:}" failed. No retries permitted until 2026-07-23 15:00:56.143100617 +0000 UTC m=+43.736551519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls") pod "dns-default-tq87x" (UID: "0a3024aa-1fca-4b32-825a-ac2b80fa4251") : secret "dns-default-metrics-tls" not found Jul 23 15:00:52.155980 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.155929 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-95l8l" event={"ID":"2df20e09-01a7-488a-aaa8-1ce22e207283","Type":"ContainerStarted","Data":"1bcacb22ca09828a0a79743b6dc32026f79a764cc6dbdf93dfdca71eaf82f3aa"} Jul 23 15:00:52.182841 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.182803 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-95l8l" podStartSLOduration=8.380214424 podStartE2EDuration="39.182791836s" totalImagesPullingTime="30.802577412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:52.182456688 +0000 UTC m=+39.775907609" watchObservedRunningTime="2026-07-23 15:00:52.182791836 +0000 UTC m=+39.776242755" Jul 23 15:00:52.211227 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.211202 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-j56c4"] Jul 23 15:00:52.219728 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.219709 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-lf7g5"] Jul 23 15:00:52.219848 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.219831 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.222425 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.222402 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 23 15:00:52.222690 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.222672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.223401 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.223375 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 23 15:00:52.223480 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.223461 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 23 15:00:52.224140 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.224121 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-jldq4\"" Jul 23 15:00:52.224538 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.224509 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 23 15:00:52.228192 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.226245 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 23 15:00:52.228192 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.226390 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 23 15:00:52.228192 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.226962 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 15:00:52.228192 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.227102 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 15:00:52.228192 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.227831 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5s4xt\"" Jul 23 15:00:52.233355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.233332 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 23 15:00:52.234605 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.234582 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-lf7g5"] Jul 23 15:00:52.234695 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.234590 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 23 15:00:52.235886 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.235862 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-j56c4"] Jul 23 15:00:52.243194 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.243175 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6fw\" (UniqueName: \"kubernetes.io/projected/bd9c8011-2021-42b4-a0f3-cafa24216bef-kube-api-access-pt6fw\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:52.243281 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.243237 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7924e63c-f6ec-4008-bc30-2d35fc835b7c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.243357 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.243277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7924e63c-f6ec-4008-bc30-2d35fc835b7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.243415 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.243368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz7rb\" (UniqueName: \"kubernetes.io/projected/7924e63c-f6ec-4008-bc30-2d35fc835b7c-kube-api-access-kz7rb\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.243480 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.243458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:52.344327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344057 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq"] Jul 23 15:00:52.344327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6fw\" (UniqueName: \"kubernetes.io/projected/bd9c8011-2021-42b4-a0f3-cafa24216bef-kube-api-access-pt6fw\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:52.344327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344219 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08654902-ac10-401e-b4a7-f3c6e5055acc-serving-cert\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.344594 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344362 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a45bae56-0ce9-4902-ba2a-604c879907cd-serving-cert\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.344594 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45bae56-0ce9-4902-ba2a-604c879907cd-tmp\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.344594 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45bae56-0ce9-4902-ba2a-604c879907cd-service-ca-bundle\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.344594 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344521 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a45bae56-0ce9-4902-ba2a-604c879907cd-snapshots\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.344594 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7924e63c-f6ec-4008-bc30-2d35fc835b7c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.344594 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7924e63c-f6ec-4008-bc30-2d35fc835b7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.344874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344660 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:52.344874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz7rb\" (UniqueName: \"kubernetes.io/projected/7924e63c-f6ec-4008-bc30-2d35fc835b7c-kube-api-access-kz7rb\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.344874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lflg\" (UniqueName: \"kubernetes.io/projected/08654902-ac10-401e-b4a7-f3c6e5055acc-kube-api-access-9lflg\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.344874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344758 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45bae56-0ce9-4902-ba2a-604c879907cd-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.344874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.344790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpzk\" (UniqueName: \"kubernetes.io/projected/a45bae56-0ce9-4902-ba2a-604c879907cd-kube-api-access-4gpzk\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.345123 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.345094 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 15:00:52.345168 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.345160 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls podName:bd9c8011-2021-42b4-a0f3-cafa24216bef nodeName:}" failed. No retries permitted until 2026-07-23 15:00:52.84514159 +0000 UTC m=+40.438592490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bdkjb" (UID: "bd9c8011-2021-42b4-a0f3-cafa24216bef") : secret "samples-operator-tls" not found Jul 23 15:00:52.345392 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.345365 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08654902-ac10-401e-b4a7-f3c6e5055acc-config\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.345509 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.345413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08654902-ac10-401e-b4a7-f3c6e5055acc-trusted-ca\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.345869 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.345842 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7924e63c-f6ec-4008-bc30-2d35fc835b7c-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.348751 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.348730 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" Jul 23 15:00:52.349857 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.349831 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7924e63c-f6ec-4008-bc30-2d35fc835b7c-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.352252 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.352224 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-s56wq\"" Jul 23 15:00:52.364544 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.364488 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq"] Jul 23 15:00:52.367142 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.367118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6fw\" (UniqueName: \"kubernetes.io/projected/bd9c8011-2021-42b4-a0f3-cafa24216bef-kube-api-access-pt6fw\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:52.370670 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.370641 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz7rb\" (UniqueName: \"kubernetes.io/projected/7924e63c-f6ec-4008-bc30-2d35fc835b7c-kube-api-access-kz7rb\") pod \"kube-storage-version-migrator-operator-7658d9b749-69l6l\" (UID: \"7924e63c-f6ec-4008-bc30-2d35fc835b7c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.413325 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.413300 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" Jul 23 15:00:52.446225 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lflg\" (UniqueName: \"kubernetes.io/projected/08654902-ac10-401e-b4a7-f3c6e5055acc-kube-api-access-9lflg\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.446337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446236 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45bae56-0ce9-4902-ba2a-604c879907cd-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.446337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpzk\" (UniqueName: \"kubernetes.io/projected/a45bae56-0ce9-4902-ba2a-604c879907cd-kube-api-access-4gpzk\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.446440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08654902-ac10-401e-b4a7-f3c6e5055acc-config\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.446440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08654902-ac10-401e-b4a7-f3c6e5055acc-trusted-ca\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.446440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446419 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08654902-ac10-401e-b4a7-f3c6e5055acc-serving-cert\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.446581 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446457 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a45bae56-0ce9-4902-ba2a-604c879907cd-serving-cert\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.446581 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446501 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45bae56-0ce9-4902-ba2a-604c879907cd-tmp\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.446677 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45bae56-0ce9-4902-ba2a-604c879907cd-service-ca-bundle\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.446677 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446620 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a45bae56-0ce9-4902-ba2a-604c879907cd-snapshots\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.446677 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.446659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrr9\" (UniqueName: \"kubernetes.io/projected/aa501884-7705-4475-97f4-d3c34e5dfa18-kube-api-access-6xrr9\") pod \"network-check-source-7bc5b6bf85-lhczq\" (UID: \"aa501884-7705-4475-97f4-d3c34e5dfa18\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" Jul 23 15:00:52.447142 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.447095 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a45bae56-0ce9-4902-ba2a-604c879907cd-tmp\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.447142 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.447124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45bae56-0ce9-4902-ba2a-604c879907cd-service-ca-bundle\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.447300 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.447220 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45bae56-0ce9-4902-ba2a-604c879907cd-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.447445 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.447423 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a45bae56-0ce9-4902-ba2a-604c879907cd-snapshots\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.447664 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.447645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/08654902-ac10-401e-b4a7-f3c6e5055acc-trusted-ca\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.447664 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.447652 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/08654902-ac10-401e-b4a7-f3c6e5055acc-config\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.448782 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.448759 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a45bae56-0ce9-4902-ba2a-604c879907cd-serving-cert\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.448857 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.448818 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/08654902-ac10-401e-b4a7-f3c6e5055acc-serving-cert\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.463028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.462987 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lflg\" (UniqueName: \"kubernetes.io/projected/08654902-ac10-401e-b4a7-f3c6e5055acc-kube-api-access-9lflg\") pod \"console-operator-565b7b569d-j56c4\" (UID: \"08654902-ac10-401e-b4a7-f3c6e5055acc\") " pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.463180 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.463162 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpzk\" (UniqueName: \"kubernetes.io/projected/a45bae56-0ce9-4902-ba2a-604c879907cd-kube-api-access-4gpzk\") pod \"insights-operator-555d9d7885-lf7g5\" (UID: \"a45bae56-0ce9-4902-ba2a-604c879907cd\") " pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.469531 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.469510 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q"] Jul 23 15:00:52.472430 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.472407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:52.475061 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.475041 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 23 15:00:52.475166 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.475068 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 23 15:00:52.475166 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.475070 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dchsf\"" Jul 23 15:00:52.485449 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.485425 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q"] Jul 23 15:00:52.541296 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.541277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:00:52.546155 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.546125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-lf7g5" Jul 23 15:00:52.547724 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.547698 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrr9\" (UniqueName: \"kubernetes.io/projected/aa501884-7705-4475-97f4-d3c34e5dfa18-kube-api-access-6xrr9\") pod \"network-check-source-7bc5b6bf85-lhczq\" (UID: \"aa501884-7705-4475-97f4-d3c34e5dfa18\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" Jul 23 15:00:52.557377 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.557357 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrr9\" (UniqueName: \"kubernetes.io/projected/aa501884-7705-4475-97f4-d3c34e5dfa18-kube-api-access-6xrr9\") pod \"network-check-source-7bc5b6bf85-lhczq\" (UID: \"aa501884-7705-4475-97f4-d3c34e5dfa18\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" Jul 23 15:00:52.593011 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.592970 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l"] Jul 23 15:00:52.595931 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:52.595861 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7924e63c_f6ec_4008_bc30_2d35fc835b7c.slice/crio-9186b67be02e7ae9a594899216380be4678279ad7f1e1fefa79ed188522cb666 WatchSource:0}: Error finding container 9186b67be02e7ae9a594899216380be4678279ad7f1e1fefa79ed188522cb666: Status 404 returned error can't find the container with id 9186b67be02e7ae9a594899216380be4678279ad7f1e1fefa79ed188522cb666 Jul 23 15:00:52.648762 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.648737 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/389cac9f-022f-4cce-a874-ef6a70b150ac-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:52.648852 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.648788 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:52.665938 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.665669 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" Jul 23 15:00:52.688552 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.688410 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-j56c4"] Jul 23 15:00:52.705112 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:52.703637 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08654902_ac10_401e_b4a7_f3c6e5055acc.slice/crio-f507e899360232fc5c1be727f983a2dcc506c25d8a1e355fda81e4c0b0cb3ec7 WatchSource:0}: Error finding container f507e899360232fc5c1be727f983a2dcc506c25d8a1e355fda81e4c0b0cb3ec7: Status 404 returned error can't find the container with id f507e899360232fc5c1be727f983a2dcc506c25d8a1e355fda81e4c0b0cb3ec7 Jul 23 15:00:52.705402 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.705123 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-lf7g5"] Jul 23 15:00:52.712848 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:52.712815 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45bae56_0ce9_4902_ba2a_604c879907cd.slice/crio-dd033985a3908041a1fe9735c778fede80d06553c289a954502a18b8c46f313f WatchSource:0}: Error finding container dd033985a3908041a1fe9735c778fede80d06553c289a954502a18b8c46f313f: Status 404 returned error can't find the container with id dd033985a3908041a1fe9735c778fede80d06553c289a954502a18b8c46f313f Jul 23 15:00:52.749498 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.749462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:52.749661 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.749643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/389cac9f-022f-4cce-a874-ef6a70b150ac-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:52.749703 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.749677 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 15:00:52.749792 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.749780 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert podName:389cac9f-022f-4cce-a874-ef6a70b150ac nodeName:}" failed. No retries permitted until 2026-07-23 15:00:53.249758572 +0000 UTC m=+40.843209484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-87q9q" (UID: "389cac9f-022f-4cce-a874-ef6a70b150ac") : secret "networking-console-plugin-cert" not found Jul 23 15:00:52.750387 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.750368 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/389cac9f-022f-4cce-a874-ef6a70b150ac-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:52.823143 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.823120 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq"] Jul 23 15:00:52.825778 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:00:52.825754 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa501884_7705_4475_97f4_d3c34e5dfa18.slice/crio-19a1ee7870c10c3a528f048ca9a5e944fbbc1ebb4a53f571010a74a45853876a WatchSource:0}: Error finding container 19a1ee7870c10c3a528f048ca9a5e944fbbc1ebb4a53f571010a74a45853876a: Status 404 returned error can't find the container with id 19a1ee7870c10c3a528f048ca9a5e944fbbc1ebb4a53f571010a74a45853876a Jul 23 15:00:52.850176 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:52.850130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:52.850295 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.850277 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 15:00:52.850347 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:52.850337 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls podName:bd9c8011-2021-42b4-a0f3-cafa24216bef nodeName:}" failed. No retries permitted until 2026-07-23 15:00:53.850320659 +0000 UTC m=+41.443771555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bdkjb" (UID: "bd9c8011-2021-42b4-a0f3-cafa24216bef") : secret "samples-operator-tls" not found Jul 23 15:00:53.159360 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.159292 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" event={"ID":"08654902-ac10-401e-b4a7-f3c6e5055acc","Type":"ContainerStarted","Data":"f507e899360232fc5c1be727f983a2dcc506c25d8a1e355fda81e4c0b0cb3ec7"} Jul 23 15:00:53.160435 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.160414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" event={"ID":"7924e63c-f6ec-4008-bc30-2d35fc835b7c","Type":"ContainerStarted","Data":"9186b67be02e7ae9a594899216380be4678279ad7f1e1fefa79ed188522cb666"} Jul 23 15:00:53.161546 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.161515 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-lf7g5" event={"ID":"a45bae56-0ce9-4902-ba2a-604c879907cd","Type":"ContainerStarted","Data":"dd033985a3908041a1fe9735c778fede80d06553c289a954502a18b8c46f313f"} Jul 23 15:00:53.162648 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.162623 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8792c" event={"ID":"1af26b3d-e245-494b-b21b-5cb771291790","Type":"ContainerStarted","Data":"5f2801dcfa7d15e01f129050c47c1d0a0efd5e197d5c7a1598c0d4d6bb771aa2"} Jul 23 15:00:53.163651 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.163632 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" event={"ID":"aa501884-7705-4475-97f4-d3c34e5dfa18","Type":"ContainerStarted","Data":"19a1ee7870c10c3a528f048ca9a5e944fbbc1ebb4a53f571010a74a45853876a"} Jul 23 15:00:53.180597 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.180558 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8792c" podStartSLOduration=2.158605477 podStartE2EDuration="5.18054463s" totalImagesPullingTime="3.021939153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:53.179893968 +0000 UTC m=+40.773344898" watchObservedRunningTime="2026-07-23 15:00:53.18054463 +0000 UTC m=+40.773995549" Jul 23 15:00:53.251973 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.251944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:53.252128 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:53.252109 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 15:00:53.252194 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:53.252183 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert podName:389cac9f-022f-4cce-a874-ef6a70b150ac nodeName:}" failed. No retries permitted until 2026-07-23 15:00:54.252163472 +0000 UTC m=+41.845614375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-87q9q" (UID: "389cac9f-022f-4cce-a874-ef6a70b150ac") : secret "networking-console-plugin-cert" not found Jul 23 15:00:53.858098 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:53.858062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:53.858341 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:53.858281 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 15:00:53.858406 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:53.858344 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls podName:bd9c8011-2021-42b4-a0f3-cafa24216bef nodeName:}" failed. No retries permitted until 2026-07-23 15:00:55.858323556 +0000 UTC m=+43.451774466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bdkjb" (UID: "bd9c8011-2021-42b4-a0f3-cafa24216bef") : secret "samples-operator-tls" not found Jul 23 15:00:54.261883 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:54.261845 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:54.262280 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:54.262148 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 15:00:54.262280 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:54.262212 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert podName:389cac9f-022f-4cce-a874-ef6a70b150ac nodeName:}" failed. No retries permitted until 2026-07-23 15:00:56.262190884 +0000 UTC m=+43.855641794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-87q9q" (UID: "389cac9f-022f-4cce-a874-ef6a70b150ac") : secret "networking-console-plugin-cert" not found Jul 23 15:00:55.875504 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:55.875467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:55.875892 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:55.875610 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 15:00:55.875892 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:55.875672 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls podName:bd9c8011-2021-42b4-a0f3-cafa24216bef nodeName:}" failed. No retries permitted until 2026-07-23 15:00:59.875657762 +0000 UTC m=+47.469108659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bdkjb" (UID: "bd9c8011-2021-42b4-a0f3-cafa24216bef") : secret "samples-operator-tls" not found Jul 23 15:00:56.077085 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:56.077049 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:00:56.077276 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:56.077214 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 15:00:56.077276 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:56.077233 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcd859484-9xr7k: secret "image-registry-tls" not found Jul 23 15:00:56.077506 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:56.077289 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls podName:a7a2dd24-8428-4072-8938-eeeb2895ecc5 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:04.077273472 +0000 UTC m=+51.670724369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls") pod "image-registry-7bcd859484-9xr7k" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5") : secret "image-registry-tls" not found Jul 23 15:00:56.177753 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:56.177675 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:00:56.177909 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:56.177840 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 15:00:56.177909 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:56.177899 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls podName:0a3024aa-1fca-4b32-825a-ac2b80fa4251 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:04.177885091 +0000 UTC m=+51.771335989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls") pod "dns-default-tq87x" (UID: "0a3024aa-1fca-4b32-825a-ac2b80fa4251") : secret "dns-default-metrics-tls" not found Jul 23 15:00:56.278335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:56.278297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:00:56.278545 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:56.278524 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 15:00:56.278635 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:56.278596 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert podName:389cac9f-022f-4cce-a874-ef6a70b150ac nodeName:}" failed. No retries permitted until 2026-07-23 15:01:00.278579471 +0000 UTC m=+47.872030386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-87q9q" (UID: "389cac9f-022f-4cce-a874-ef6a70b150ac") : secret "networking-console-plugin-cert" not found Jul 23 15:00:59.182151 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.182023 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" event={"ID":"aa501884-7705-4475-97f4-d3c34e5dfa18","Type":"ContainerStarted","Data":"e6c2b28a8a20b4ac736166660e1574aa3db1e571e79b4c6d520668ccd6def3c8"} Jul 23 15:00:59.183811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.183790 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/0.log" Jul 23 15:00:59.183936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.183829 2568 generic.go:356] "Generic (PLEG): container finished" podID="08654902-ac10-401e-b4a7-f3c6e5055acc" containerID="4d76f5ed89a747e752584f1de6caeb258d7627a059342346475d2f17c72d6f64" exitCode=255 Jul 23 15:00:59.183936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.183904 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" event={"ID":"08654902-ac10-401e-b4a7-f3c6e5055acc","Type":"ContainerDied","Data":"4d76f5ed89a747e752584f1de6caeb258d7627a059342346475d2f17c72d6f64"} Jul 23 15:00:59.184126 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.184109 2568 scope.go:117] "RemoveContainer" containerID="4d76f5ed89a747e752584f1de6caeb258d7627a059342346475d2f17c72d6f64" Jul 23 15:00:59.185573 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.185530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" event={"ID":"7924e63c-f6ec-4008-bc30-2d35fc835b7c","Type":"ContainerStarted","Data":"35ec9f84f91f7e90deb86916dda18243b243d1e1b87151415e6b43c13333d0ae"} Jul 23 15:00:59.190103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.190080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-lf7g5" event={"ID":"a45bae56-0ce9-4902-ba2a-604c879907cd","Type":"ContainerStarted","Data":"a0bb54fc9c3313b8a15a4cc2e75099c5ff2eef4ed466a84fd1c7b7768f243e09"} Jul 23 15:00:59.214448 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.214404 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-lhczq" podStartSLOduration=1.849545196 podStartE2EDuration="7.214387544s" totalImagesPullingTime="5.364842348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:59.211602218 +0000 UTC m=+46.805053138" watchObservedRunningTime="2026-07-23 15:00:59.214387544 +0000 UTC m=+46.807838461" Jul 23 15:00:59.246816 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.246776 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-lf7g5" podStartSLOduration=1.7702419109999998 podStartE2EDuration="7.246760149s" totalImagesPullingTime="5.476518238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:59.246025529 +0000 UTC m=+46.839476441" watchObservedRunningTime="2026-07-23 15:00:59.246760149 +0000 UTC m=+46.840211060" Jul 23 15:00:59.281384 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.280277 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" podStartSLOduration=1.687329225 podStartE2EDuration="7.280261518s" totalImagesPullingTime="5.592932293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:00:59.279637817 +0000 UTC m=+46.873088737" watchObservedRunningTime="2026-07-23 15:00:59.280261518 +0000 UTC m=+46.873712438" Jul 23 15:00:59.655766 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.655736 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lvr6j"] Jul 23 15:00:59.673975 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.673944 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.674531 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.674507 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lvr6j"] Jul 23 15:00:59.677372 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.677352 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 15:00:59.677473 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.677406 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5f6ms\"" Jul 23 15:00:59.682304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.682280 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 15:00:59.707540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.707520 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcd2b199-7aac-46f3-a9fc-e7514d889860-crio-socket\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.707644 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.707564 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcd2b199-7aac-46f3-a9fc-e7514d889860-data-volume\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.707644 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.707600 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.707780 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.707664 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4z42\" (UniqueName: \"kubernetes.io/projected/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-api-access-v4z42\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.707780 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.707741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.808912 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.808861 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.809119 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.808948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcd2b199-7aac-46f3-a9fc-e7514d889860-crio-socket\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.809119 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:59.808978 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 15:00:59.809119 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.809029 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcd2b199-7aac-46f3-a9fc-e7514d889860-data-volume\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.809119 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:59.809076 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:00.309060614 +0000 UTC m=+47.902511533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : configmap "kube-rbac-proxy" not found Jul 23 15:00:59.809119 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.809103 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.809401 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.809187 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fcd2b199-7aac-46f3-a9fc-e7514d889860-crio-socket\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.809401 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.809195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4z42\" (UniqueName: \"kubernetes.io/projected/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-api-access-v4z42\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.809401 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:59.809282 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 15:00:59.809401 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:59.809365 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:00.309342518 +0000 UTC m=+47.902793446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : secret "insights-runtime-extractor-tls" not found Jul 23 15:00:59.809534 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.809418 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcd2b199-7aac-46f3-a9fc-e7514d889860-data-volume\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.822572 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.822545 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4z42\" (UniqueName: \"kubernetes.io/projected/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-api-access-v4z42\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:00:59.910404 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:00:59.910334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:00:59.910524 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:59.910449 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 15:00:59.910524 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:00:59.910506 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls podName:bd9c8011-2021-42b4-a0f3-cafa24216bef nodeName:}" failed. No retries permitted until 2026-07-23 15:01:07.910490739 +0000 UTC m=+55.503941639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bdkjb" (UID: "bd9c8011-2021-42b4-a0f3-cafa24216bef") : secret "samples-operator-tls" not found Jul 23 15:01:00.194711 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.194683 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:01:00.195103 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.195088 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/0.log" Jul 23 15:01:00.195155 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.195135 2568 generic.go:356] "Generic (PLEG): container finished" podID="08654902-ac10-401e-b4a7-f3c6e5055acc" containerID="b05f165a1eb0ee0068500c8e8774b37ebbced9f8002810bf192ac77a1b352124" exitCode=255 Jul 23 15:01:00.195219 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.195205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" event={"ID":"08654902-ac10-401e-b4a7-f3c6e5055acc","Type":"ContainerDied","Data":"b05f165a1eb0ee0068500c8e8774b37ebbced9f8002810bf192ac77a1b352124"} Jul 23 15:01:00.195272 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.195262 2568 scope.go:117] "RemoveContainer" containerID="4d76f5ed89a747e752584f1de6caeb258d7627a059342346475d2f17c72d6f64" Jul 23 15:01:00.195480 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.195461 2568 scope.go:117] "RemoveContainer" containerID="b05f165a1eb0ee0068500c8e8774b37ebbced9f8002810bf192ac77a1b352124" Jul 23 15:01:00.195688 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:00.195666 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-j56c4_openshift-console-operator(08654902-ac10-401e-b4a7-f3c6e5055acc)\"" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" podUID="08654902-ac10-401e-b4a7-f3c6e5055acc" Jul 23 15:01:00.314056 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.314019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:00.314149 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.314070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:01:00.314205 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:00.314148 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 15:01:00.314205 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:00.314154 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 15:01:00.314205 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:00.314202 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:01.314188041 +0000 UTC m=+48.907638939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : secret "insights-runtime-extractor-tls" not found Jul 23 15:01:00.314316 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:00.314226 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert podName:389cac9f-022f-4cce-a874-ef6a70b150ac nodeName:}" failed. No retries permitted until 2026-07-23 15:01:08.314219996 +0000 UTC m=+55.907670892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-87q9q" (UID: "389cac9f-022f-4cce-a874-ef6a70b150ac") : secret "networking-console-plugin-cert" not found Jul 23 15:01:00.314316 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:00.314279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:00.314399 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:00.314378 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 15:01:00.314444 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:00.314435 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:01.314421549 +0000 UTC m=+48.907872446 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : configmap "kube-rbac-proxy" not found Jul 23 15:01:01.199619 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:01.199595 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:01:01.199986 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:01.199878 2568 scope.go:117] "RemoveContainer" containerID="b05f165a1eb0ee0068500c8e8774b37ebbced9f8002810bf192ac77a1b352124" Jul 23 15:01:01.200065 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:01.200027 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-j56c4_openshift-console-operator(08654902-ac10-401e-b4a7-f3c6e5055acc)\"" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" podUID="08654902-ac10-401e-b4a7-f3c6e5055acc" Jul 23 15:01:01.320243 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:01.320205 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:01.320383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:01.320275 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:01.320383 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:01.320325 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 15:01:01.320383 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:01.320378 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 15:01:01.320481 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:01.320380 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:03.320366548 +0000 UTC m=+50.913817445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : configmap "kube-rbac-proxy" not found Jul 23 15:01:01.320481 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:01.320420 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:03.320410196 +0000 UTC m=+50.913861093 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : secret "insights-runtime-extractor-tls" not found Jul 23 15:01:01.431815 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:01.431788 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2tnf9_0b36d684-b669-45b1-b436-4b2d52a6134a/dns-node-resolver/0.log" Jul 23 15:01:02.541420 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:02.541389 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:01:02.541777 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:02.541474 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:01:02.541819 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:02.541792 2568 scope.go:117] "RemoveContainer" containerID="b05f165a1eb0ee0068500c8e8774b37ebbced9f8002810bf192ac77a1b352124" Jul 23 15:01:02.541944 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:02.541926 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-j56c4_openshift-console-operator(08654902-ac10-401e-b4a7-f3c6e5055acc)\"" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" podUID="08654902-ac10-401e-b4a7-f3c6e5055acc" Jul 23 15:01:02.834886 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:02.834814 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vwksz_66136ccc-448f-4364-83e7-df4ba4703e61/node-ca/0.log" Jul 23 15:01:03.336113 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:03.336077 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:03.336282 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:03.336143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:03.336282 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:03.336190 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 23 15:01:03.336282 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:03.336254 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:07.336238595 +0000 UTC m=+54.929689492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : configmap "kube-rbac-proxy" not found Jul 23 15:01:03.336282 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:03.336259 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 15:01:03.336461 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:03.336298 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:07.336287664 +0000 UTC m=+54.929738561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : secret "insights-runtime-extractor-tls" not found Jul 23 15:01:03.432590 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:03.432568 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8792c_1af26b3d-e245-494b-b21b-5cb771291790/serve-healthcheck-canary/0.log" Jul 23 15:01:04.142514 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:04.142478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:01:04.142925 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:04.142632 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 23 15:01:04.142925 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:04.142653 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bcd859484-9xr7k: secret "image-registry-tls" not found Jul 23 15:01:04.142925 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:04.142711 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls podName:a7a2dd24-8428-4072-8938-eeeb2895ecc5 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:20.142694299 +0000 UTC m=+67.736145196 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls") pod "image-registry-7bcd859484-9xr7k" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5") : secret "image-registry-tls" not found Jul 23 15:01:04.243520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:04.243494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:01:04.243651 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:04.243594 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 23 15:01:04.243651 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:04.243639 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls podName:0a3024aa-1fca-4b32-825a-ac2b80fa4251 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:20.243627896 +0000 UTC m=+67.837078793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls") pod "dns-default-tq87x" (UID: "0a3024aa-1fca-4b32-825a-ac2b80fa4251") : secret "dns-default-metrics-tls" not found Jul 23 15:01:07.367171 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:07.367133 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:07.367633 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:07.367208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:07.367633 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:07.367315 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 23 15:01:07.367633 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:07.367374 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls podName:fcd2b199-7aac-46f3-a9fc-e7514d889860 nodeName:}" failed. No retries permitted until 2026-07-23 15:01:15.367357661 +0000 UTC m=+62.960808575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lvr6j" (UID: "fcd2b199-7aac-46f3-a9fc-e7514d889860") : secret "insights-runtime-extractor-tls" not found Jul 23 15:01:07.367633 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:07.367608 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fcd2b199-7aac-46f3-a9fc-e7514d889860-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:07.970718 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:07.970690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:01:07.970889 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:07.970834 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 23 15:01:07.970941 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:07.970894 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls podName:bd9c8011-2021-42b4-a0f3-cafa24216bef nodeName:}" failed. No retries permitted until 2026-07-23 15:01:23.970880061 +0000 UTC m=+71.564330959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-bdkjb" (UID: "bd9c8011-2021-42b4-a0f3-cafa24216bef") : secret "samples-operator-tls" not found Jul 23 15:01:08.374572 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:08.374493 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:01:08.374959 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:08.374613 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 23 15:01:08.374959 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:08.374666 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert podName:389cac9f-022f-4cce-a874-ef6a70b150ac nodeName:}" failed. No retries permitted until 2026-07-23 15:01:24.374652989 +0000 UTC m=+71.968103886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-87q9q" (UID: "389cac9f-022f-4cce-a874-ef6a70b150ac") : secret "networking-console-plugin-cert" not found Jul 23 15:01:14.134607 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:14.134578 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-66ftl" Jul 23 15:01:15.431072 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:15.431037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:15.433222 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:15.433189 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fcd2b199-7aac-46f3-a9fc-e7514d889860-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lvr6j\" (UID: \"fcd2b199-7aac-46f3-a9fc-e7514d889860\") " pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:15.587160 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:15.587133 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5f6ms\"" Jul 23 15:01:15.594650 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:15.594633 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lvr6j" Jul 23 15:01:15.716270 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:15.716127 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lvr6j"] Jul 23 15:01:15.718853 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:15.718830 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd2b199_7aac_46f3_a9fc_e7514d889860.slice/crio-66fe9efa59c5b62f182c63142d83867977b8aad165d8dcd8c79fc592f927e02a WatchSource:0}: Error finding container 66fe9efa59c5b62f182c63142d83867977b8aad165d8dcd8c79fc592f927e02a: Status 404 returned error can't find the container with id 66fe9efa59c5b62f182c63142d83867977b8aad165d8dcd8c79fc592f927e02a Jul 23 15:01:15.959446 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:15.959414 2568 scope.go:117] "RemoveContainer" containerID="b05f165a1eb0ee0068500c8e8774b37ebbced9f8002810bf192ac77a1b352124" Jul 23 15:01:16.245173 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:16.245083 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lvr6j" event={"ID":"fcd2b199-7aac-46f3-a9fc-e7514d889860","Type":"ContainerStarted","Data":"66fe9efa59c5b62f182c63142d83867977b8aad165d8dcd8c79fc592f927e02a"} Jul 23 15:01:16.246874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:16.246854 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:01:16.247017 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:16.246901 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" event={"ID":"08654902-ac10-401e-b4a7-f3c6e5055acc","Type":"ContainerStarted","Data":"a6257d82faf88530d2ec7acd58049abbab521529d678150c022cc54e38a19f72"} Jul 23 15:01:16.247193 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:16.247175 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:01:17.247382 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:17.247334 2568 patch_prober.go:28] interesting pod/console-operator-565b7b569d-j56c4 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 15:01:17.247806 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:17.247406 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" podUID="08654902-ac10-401e-b4a7-f3c6e5055acc" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 23 15:01:17.396186 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:17.396157 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" Jul 23 15:01:17.419395 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:17.419340 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-j56c4" podStartSLOduration=19.928039397 podStartE2EDuration="25.419324568s" totalImagesPullingTime="5.491285171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:16.265540787 +0000 UTC m=+63.858991706" watchObservedRunningTime="2026-07-23 15:01:17.419324568 +0000 UTC m=+65.012775486" Jul 23 15:01:18.658315 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.658275 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:01:18.661526 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.661505 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 15:01:18.671607 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.671580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00c24181-cecc-4735-aa71-d217f7dcc3b1-original-pull-secret\") pod \"global-pull-secret-syncer-g7spv\" (UID: \"00c24181-cecc-4735-aa71-d217f7dcc3b1\") " pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:01:18.759305 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.759277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:01:18.762682 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.762657 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 15:01:18.772203 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.772176 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/954165e8-54ba-4595-b41a-186d83bd2818-metrics-certs\") pod \"network-metrics-daemon-f6mzs\" (UID: \"954165e8-54ba-4595-b41a-186d83bd2818\") " pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:01:18.773018 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.772987 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7spv" Jul 23 15:01:18.860672 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.860644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:01:18.863469 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.863443 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxcws\" (UniqueName: \"kubernetes.io/projected/e4b67358-0326-47cb-bb64-03ba0411673d-kube-api-access-lxcws\") pod \"network-check-target-6cr64\" (UID: \"e4b67358-0326-47cb-bb64-03ba0411673d\") " pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:01:18.971679 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.971655 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qbp9q\"" Jul 23 15:01:18.978957 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:18.978938 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f6mzs" Jul 23 15:01:19.071827 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.071799 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v5f82\"" Jul 23 15:01:19.079355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.079332 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:01:19.792473 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.792438 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf"] Jul 23 15:01:19.804778 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.804253 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.810904 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.810717 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 15:01:19.811384 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.811349 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf"] Jul 23 15:01:19.811921 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.811880 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 15:01:19.812095 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.811944 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 23 15:01:19.815220 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.815196 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 15:01:19.868719 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.868690 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/885720dc-d919-4224-bbee-31fa3aa3c0a2-klusterlet-config\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.868874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.868742 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbdf\" (UniqueName: \"kubernetes.io/projected/885720dc-d919-4224-bbee-31fa3aa3c0a2-kube-api-access-bzbdf\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.868874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.868781 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/885720dc-d919-4224-bbee-31fa3aa3c0a2-tmp\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.877832 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.877802 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-dgg29"] Jul 23 15:01:19.883060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.883035 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-dgg29" Jul 23 15:01:19.886513 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.886492 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 15:01:19.886857 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.886841 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 15:01:19.887254 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.887238 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-5c75q\"" Jul 23 15:01:19.903366 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.903346 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-dgg29"] Jul 23 15:01:19.970102 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.970069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/885720dc-d919-4224-bbee-31fa3aa3c0a2-klusterlet-config\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.970281 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.970136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbdf\" (UniqueName: \"kubernetes.io/projected/885720dc-d919-4224-bbee-31fa3aa3c0a2-kube-api-access-bzbdf\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.970281 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.970170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ch5t\" (UniqueName: \"kubernetes.io/projected/c42d3923-b5f6-48c5-913d-1aec82494188-kube-api-access-8ch5t\") pod \"downloads-5bc67dcd46-dgg29\" (UID: \"c42d3923-b5f6-48c5-913d-1aec82494188\") " pod="openshift-console/downloads-5bc67dcd46-dgg29" Jul 23 15:01:19.970281 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.970225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/885720dc-d919-4224-bbee-31fa3aa3c0a2-tmp\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.970617 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.970592 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/885720dc-d919-4224-bbee-31fa3aa3c0a2-tmp\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.972867 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.972824 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/885720dc-d919-4224-bbee-31fa3aa3c0a2-klusterlet-config\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:19.988414 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:19.988388 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbdf\" (UniqueName: \"kubernetes.io/projected/885720dc-d919-4224-bbee-31fa3aa3c0a2-kube-api-access-bzbdf\") pod \"klusterlet-addon-workmgr-757c4db7f4-d4qjf\" (UID: \"885720dc-d919-4224-bbee-31fa3aa3c0a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:20.071201 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.071122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ch5t\" (UniqueName: \"kubernetes.io/projected/c42d3923-b5f6-48c5-913d-1aec82494188-kube-api-access-8ch5t\") pod \"downloads-5bc67dcd46-dgg29\" (UID: \"c42d3923-b5f6-48c5-913d-1aec82494188\") " pod="openshift-console/downloads-5bc67dcd46-dgg29" Jul 23 15:01:20.098698 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.098667 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ch5t\" (UniqueName: \"kubernetes.io/projected/c42d3923-b5f6-48c5-913d-1aec82494188-kube-api-access-8ch5t\") pod \"downloads-5bc67dcd46-dgg29\" (UID: \"c42d3923-b5f6-48c5-913d-1aec82494188\") " pod="openshift-console/downloads-5bc67dcd46-dgg29" Jul 23 15:01:20.117411 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.117381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:20.171686 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.171659 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:01:20.174349 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.174329 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"image-registry-7bcd859484-9xr7k\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:01:20.193257 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.193234 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-dgg29" Jul 23 15:01:20.272451 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.272427 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:01:20.275146 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.275120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a3024aa-1fca-4b32-825a-ac2b80fa4251-metrics-tls\") pod \"dns-default-tq87x\" (UID: \"0a3024aa-1fca-4b32-825a-ac2b80fa4251\") " pod="openshift-dns/dns-default-tq87x" Jul 23 15:01:20.371945 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.371882 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qpm5s\"" Jul 23 15:01:20.380058 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.380040 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:01:20.493897 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.493868 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-msfpg\"" Jul 23 15:01:20.501649 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.501628 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tq87x" Jul 23 15:01:20.915762 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:20.915720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf"] Jul 23 15:01:21.142066 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:21.141968 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885720dc_d919_4224_bbee_31fa3aa3c0a2.slice/crio-46b006c83df54adda53c549d93d37164750e3b833a3a5bbc88842f4f1e52445e WatchSource:0}: Error finding container 46b006c83df54adda53c549d93d37164750e3b833a3a5bbc88842f4f1e52445e: Status 404 returned error can't find the container with id 46b006c83df54adda53c549d93d37164750e3b833a3a5bbc88842f4f1e52445e Jul 23 15:01:21.271800 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:21.271729 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" event={"ID":"885720dc-d919-4224-bbee-31fa3aa3c0a2","Type":"ContainerStarted","Data":"46b006c83df54adda53c549d93d37164750e3b833a3a5bbc88842f4f1e52445e"} Jul 23 15:01:21.360040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:21.359942 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bcd859484-9xr7k"] Jul 23 15:01:21.364585 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:21.364539 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a2dd24_8428_4072_8938_eeeb2895ecc5.slice/crio-4fa549ac6d085eba80a3032bf261b41417af7fe7831c99c11bb264c541e3deca WatchSource:0}: Error finding container 4fa549ac6d085eba80a3032bf261b41417af7fe7831c99c11bb264c541e3deca: Status 404 returned error can't find the container with id 4fa549ac6d085eba80a3032bf261b41417af7fe7831c99c11bb264c541e3deca Jul 23 15:01:21.365629 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:21.365574 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tq87x"] Jul 23 15:01:21.368768 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:21.368632 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3024aa_1fca_4b32_825a_ac2b80fa4251.slice/crio-4963519954b6abc31390131577fa16c6b18b5f12348f8aecb6eabca632a78490 WatchSource:0}: Error finding container 4963519954b6abc31390131577fa16c6b18b5f12348f8aecb6eabca632a78490: Status 404 returned error can't find the container with id 4963519954b6abc31390131577fa16c6b18b5f12348f8aecb6eabca632a78490 Jul 23 15:01:21.378261 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:21.378236 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-dgg29"] Jul 23 15:01:21.381510 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:21.381476 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42d3923_b5f6_48c5_913d_1aec82494188.slice/crio-46738a344f411f42f413dd05812606ba338ee4e3a8b91c8f8e135787f518d24e WatchSource:0}: Error finding container 46738a344f411f42f413dd05812606ba338ee4e3a8b91c8f8e135787f518d24e: Status 404 returned error can't find the container with id 46738a344f411f42f413dd05812606ba338ee4e3a8b91c8f8e135787f518d24e Jul 23 15:01:21.605958 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:21.605932 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7spv"] Jul 23 15:01:21.609417 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:21.609380 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f6mzs"] Jul 23 15:01:21.610653 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:21.610471 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6cr64"] Jul 23 15:01:21.612818 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:21.612798 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b67358_0326_47cb_bb64_03ba0411673d.slice/crio-a22e96e2d4d00074dd78daf1f7919e4801dbeb73277c10a3519e3c9afa888917 WatchSource:0}: Error finding container a22e96e2d4d00074dd78daf1f7919e4801dbeb73277c10a3519e3c9afa888917: Status 404 returned error can't find the container with id a22e96e2d4d00074dd78daf1f7919e4801dbeb73277c10a3519e3c9afa888917 Jul 23 15:01:21.614074 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:21.614054 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954165e8_54ba_4595_b41a_186d83bd2818.slice/crio-8911a321569ffb2363b9ced7492a4ed3c6bc6b5f520b73a9ca217efdfca1e650 WatchSource:0}: Error finding container 8911a321569ffb2363b9ced7492a4ed3c6bc6b5f520b73a9ca217efdfca1e650: Status 404 returned error can't find the container with id 8911a321569ffb2363b9ced7492a4ed3c6bc6b5f520b73a9ca217efdfca1e650 Jul 23 15:01:22.288724 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.288632 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lvr6j" event={"ID":"fcd2b199-7aac-46f3-a9fc-e7514d889860","Type":"ContainerStarted","Data":"c33ef7b98aa24bf9c3c32d9eb09388accf206c53e5ce253c5d15f8765c92562b"} Jul 23 15:01:22.291360 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.291262 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-dgg29" event={"ID":"c42d3923-b5f6-48c5-913d-1aec82494188","Type":"ContainerStarted","Data":"46738a344f411f42f413dd05812606ba338ee4e3a8b91c8f8e135787f518d24e"} Jul 23 15:01:22.294160 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.294133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6mzs" event={"ID":"954165e8-54ba-4595-b41a-186d83bd2818","Type":"ContainerStarted","Data":"8911a321569ffb2363b9ced7492a4ed3c6bc6b5f520b73a9ca217efdfca1e650"} Jul 23 15:01:22.296532 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.296452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tq87x" event={"ID":"0a3024aa-1fca-4b32-825a-ac2b80fa4251","Type":"ContainerStarted","Data":"4963519954b6abc31390131577fa16c6b18b5f12348f8aecb6eabca632a78490"} Jul 23 15:01:22.301343 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.300491 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" event={"ID":"a7a2dd24-8428-4072-8938-eeeb2895ecc5","Type":"ContainerStarted","Data":"e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5"} Jul 23 15:01:22.301343 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.300535 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" event={"ID":"a7a2dd24-8428-4072-8938-eeeb2895ecc5","Type":"ContainerStarted","Data":"4fa549ac6d085eba80a3032bf261b41417af7fe7831c99c11bb264c541e3deca"} Jul 23 15:01:22.301343 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.301300 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:01:22.304703 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.304050 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6cr64" event={"ID":"e4b67358-0326-47cb-bb64-03ba0411673d","Type":"ContainerStarted","Data":"4bf5217f8cb63e635acdd60bc3332bdc618e35f58a42b7273ca51e6c4c5cc083"} Jul 23 15:01:22.304703 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.304091 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6cr64" event={"ID":"e4b67358-0326-47cb-bb64-03ba0411673d","Type":"ContainerStarted","Data":"a22e96e2d4d00074dd78daf1f7919e4801dbeb73277c10a3519e3c9afa888917"} Jul 23 15:01:22.304703 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.304492 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:01:22.306614 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.306532 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7spv" event={"ID":"00c24181-cecc-4735-aa71-d217f7dcc3b1","Type":"ContainerStarted","Data":"1a51820819f74e295df0c5027c9f86ca3b59ad98caf1a802654ff75f315cff43"} Jul 23 15:01:22.346461 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.346407 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6cr64" podStartSLOduration=69.346388625 podStartE2EDuration="1m9.346388625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:22.345113177 +0000 UTC m=+69.938564097" watchObservedRunningTime="2026-07-23 15:01:22.346388625 +0000 UTC m=+69.939839548" Jul 23 15:01:22.346585 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:22.346515 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" podStartSLOduration=69.346506483 podStartE2EDuration="1m9.346506483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:22.327066168 +0000 UTC m=+69.920517088" watchObservedRunningTime="2026-07-23 15:01:22.346506483 +0000 UTC m=+69.939957405" Jul 23 15:01:24.008978 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.008935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:01:24.021031 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.020953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/bd9c8011-2021-42b4-a0f3-cafa24216bef-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-bdkjb\" (UID: \"bd9c8011-2021-42b4-a0f3-cafa24216bef\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:01:24.228015 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.227818 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6fqx9\"" Jul 23 15:01:24.232185 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.231984 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" Jul 23 15:01:24.414547 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.413503 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:01:24.419509 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.419464 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/389cac9f-022f-4cce-a874-ef6a70b150ac-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-87q9q\" (UID: \"389cac9f-022f-4cce-a874-ef6a70b150ac\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:01:24.584634 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.584592 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dchsf\"" Jul 23 15:01:24.592259 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:24.592234 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" Jul 23 15:01:30.963452 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:30.963423 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q"] Jul 23 15:01:31.165841 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.165772 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb"] Jul 23 15:01:31.348852 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.348746 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7spv" event={"ID":"00c24181-cecc-4735-aa71-d217f7dcc3b1","Type":"ContainerStarted","Data":"d11f803fc247b2d53bf682cb93c8732a0cf146ba4edeb7f531287f3a208c4274"} Jul 23 15:01:31.350508 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.350449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" event={"ID":"bd9c8011-2021-42b4-a0f3-cafa24216bef","Type":"ContainerStarted","Data":"bf29101350485fd9feaf2b89b76f1714042a2acdacf5c51816c5423810050ac0"} Jul 23 15:01:31.354592 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.354568 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" event={"ID":"389cac9f-022f-4cce-a874-ef6a70b150ac","Type":"ContainerStarted","Data":"b9ad338f978e57bd7c1b726cd033ec869c5b372cea52082a04da2467b71c30f9"} Jul 23 15:01:31.356756 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.356314 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" event={"ID":"885720dc-d919-4224-bbee-31fa3aa3c0a2","Type":"ContainerStarted","Data":"1eb233f80891666ef57ef51dc97afa7c03cc8afe6fc3c6f6babe965a73b4639e"} Jul 23 15:01:31.356864 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.356840 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:31.358663 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.358625 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" Jul 23 15:01:31.359143 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.359110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6mzs" event={"ID":"954165e8-54ba-4595-b41a-186d83bd2818","Type":"ContainerStarted","Data":"d3d20912026ba5b19f28b3d8bf883b3c971b3d563c56b97df7e22b3712edbd63"} Jul 23 15:01:31.359237 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.359178 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f6mzs" event={"ID":"954165e8-54ba-4595-b41a-186d83bd2818","Type":"ContainerStarted","Data":"0ce027a37f2ef5e45426a90a2ba30d374539efd00f673f7241d11fc2e4911a64"} Jul 23 15:01:31.361091 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.361030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tq87x" event={"ID":"0a3024aa-1fca-4b32-825a-ac2b80fa4251","Type":"ContainerStarted","Data":"f77c72c9b1a6fe7b4ab6d2333dabfc43c0474e77b048c2ce41c558744ebd668e"} Jul 23 15:01:31.383579 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.383529 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-757c4db7f4-d4qjf" podStartSLOduration=2.516774232 podStartE2EDuration="12.383512018s" totalImagesPullingTime="9.866737786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:31.38347146 +0000 UTC m=+78.976922380" watchObservedRunningTime="2026-07-23 15:01:31.383512018 +0000 UTC m=+78.976962941" Jul 23 15:01:31.383693 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.383647 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g7spv" podStartSLOduration=68.977728628 podStartE2EDuration="1m18.383638466s" totalImagesPullingTime="9.405909838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:31.366433316 +0000 UTC m=+78.959884248" watchObservedRunningTime="2026-07-23 15:01:31.383638466 +0000 UTC m=+78.977089384" Jul 23 15:01:31.403468 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:31.403427 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f6mzs" podStartSLOduration=69.997379544 podStartE2EDuration="1m18.403417845s" totalImagesPullingTime="8.406038301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:31.402689554 +0000 UTC m=+78.996140473" watchObservedRunningTime="2026-07-23 15:01:31.403417845 +0000 UTC m=+78.996868763" Jul 23 15:01:32.074683 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.074647 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2"] Jul 23 15:01:32.103580 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.103511 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:32.107239 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.107213 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lftrq\"" Jul 23 15:01:32.108091 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.108067 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 15:01:32.108788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.108730 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2"] Jul 23 15:01:32.181351 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.181319 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-pcbm2\" (UID: \"3d4deab9-b559-49d6-a1eb-d21d6f9111bd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:32.282860 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.282798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-pcbm2\" (UID: \"3d4deab9-b559-49d6-a1eb-d21d6f9111bd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:32.283047 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:32.282937 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 23 15:01:32.283288 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:32.283262 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates podName:3d4deab9-b559-49d6-a1eb-d21d6f9111bd nodeName:}" failed. No retries permitted until 2026-07-23 15:01:32.7832342 +0000 UTC m=+80.376685117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-pcbm2" (UID: "3d4deab9-b559-49d6-a1eb-d21d6f9111bd") : secret "prometheus-operator-admission-webhook-tls" not found Jul 23 15:01:32.368659 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.368519 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lvr6j" event={"ID":"fcd2b199-7aac-46f3-a9fc-e7514d889860","Type":"ContainerStarted","Data":"87cfad7509d3bf28a0087d7baaf88027ca4bce43e546d278bda6dd357f92fd88"} Jul 23 15:01:32.371005 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.370960 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tq87x" event={"ID":"0a3024aa-1fca-4b32-825a-ac2b80fa4251","Type":"ContainerStarted","Data":"42e6dec4d93a88147afd072405379054161cc7d7533a37ae804f4489b7d95d8e"} Jul 23 15:01:32.371744 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.371620 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tq87x" Jul 23 15:01:32.416251 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.415817 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tq87x" podStartSLOduration=34.77165663 podStartE2EDuration="44.415796037s" totalImagesPullingTime="9.644139407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:32.41218851 +0000 UTC m=+80.005639429" watchObservedRunningTime="2026-07-23 15:01:32.415796037 +0000 UTC m=+80.009246957" Jul 23 15:01:32.786330 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:32.786298 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-pcbm2\" (UID: \"3d4deab9-b559-49d6-a1eb-d21d6f9111bd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:32.786544 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:32.786465 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 23 15:01:32.786544 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:32.786519 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates podName:3d4deab9-b559-49d6-a1eb-d21d6f9111bd nodeName:}" failed. No retries permitted until 2026-07-23 15:01:33.786505596 +0000 UTC m=+81.379956494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-pcbm2" (UID: "3d4deab9-b559-49d6-a1eb-d21d6f9111bd") : secret "prometheus-operator-admission-webhook-tls" not found Jul 23 15:01:33.379418 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:33.379372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" event={"ID":"389cac9f-022f-4cce-a874-ef6a70b150ac","Type":"ContainerStarted","Data":"956fc47e8ee1191cb124eeee25bc93da1387f0469a69cb66ca12ea42a16ed85d"} Jul 23 15:01:33.794352 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:33.794323 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-pcbm2\" (UID: \"3d4deab9-b559-49d6-a1eb-d21d6f9111bd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:33.797388 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:33.797328 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3d4deab9-b559-49d6-a1eb-d21d6f9111bd-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-pcbm2\" (UID: \"3d4deab9-b559-49d6-a1eb-d21d6f9111bd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:33.919091 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:33.918299 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:34.093732 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:34.093587 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-87q9q" podStartSLOduration=40.333579162 podStartE2EDuration="42.093562823s" totalImagesPullingTime="1.759983661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:33.409429228 +0000 UTC m=+81.002880151" watchObservedRunningTime="2026-07-23 15:01:34.093562823 +0000 UTC m=+81.687013744" Jul 23 15:01:34.093908 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:34.093880 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2"] Jul 23 15:01:34.097240 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:34.097209 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4deab9_b559_49d6_a1eb_d21d6f9111bd.slice/crio-b45fd520aacbf1732faa64dabc68b642faf551ef6865b18305644672f2b6623e WatchSource:0}: Error finding container b45fd520aacbf1732faa64dabc68b642faf551ef6865b18305644672f2b6623e: Status 404 returned error can't find the container with id b45fd520aacbf1732faa64dabc68b642faf551ef6865b18305644672f2b6623e Jul 23 15:01:34.386275 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:34.386185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" event={"ID":"bd9c8011-2021-42b4-a0f3-cafa24216bef","Type":"ContainerStarted","Data":"5aecf06c93002b08ee5758216cbbdc26bb6e24776e7c8fc3c78e39ff93160c34"} Jul 23 15:01:34.386275 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:34.386246 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" event={"ID":"bd9c8011-2021-42b4-a0f3-cafa24216bef","Type":"ContainerStarted","Data":"eb19606a9572408f89be8bcaecd7ac4773f1e627ae9a40e9e79a96217edab4fc"} Jul 23 15:01:34.387770 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:34.387717 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" event={"ID":"3d4deab9-b559-49d6-a1eb-d21d6f9111bd","Type":"ContainerStarted","Data":"b45fd520aacbf1732faa64dabc68b642faf551ef6865b18305644672f2b6623e"} Jul 23 15:01:34.405936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:34.405868 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-bdkjb" podStartSLOduration=40.047012257 podStartE2EDuration="42.405850393s" totalImagesPullingTime="2.358838136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:34.405500338 +0000 UTC m=+81.998951256" watchObservedRunningTime="2026-07-23 15:01:34.405850393 +0000 UTC m=+81.999301313" Jul 23 15:01:42.383340 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:42.383211 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tq87x" Jul 23 15:01:43.417986 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:43.417948 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bcd859484-9xr7k"] Jul 23 15:01:43.422625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:43.422600 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:01:44.432335 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:44.432295 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lvr6j" event={"ID":"fcd2b199-7aac-46f3-a9fc-e7514d889860","Type":"ContainerStarted","Data":"ded2c1196859469fc940e3e849bbd4ce1275793ff4291db61835336012088dee"} Jul 23 15:01:44.435176 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:44.435145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-dgg29" event={"ID":"c42d3923-b5f6-48c5-913d-1aec82494188","Type":"ContainerStarted","Data":"fd1376b4fdeabdff6bc8644d3923989ef0e1686cceb2d5cbf9a28112b79a75cc"} Jul 23 15:01:44.435545 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:44.435515 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-dgg29" Jul 23 15:01:44.449068 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:44.449015 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-dgg29" Jul 23 15:01:44.455522 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:44.455422 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lvr6j" podStartSLOduration=17.682807173 podStartE2EDuration="45.455402462s" totalImagesPullingTime="27.772595289s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:00:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:44.455278856 +0000 UTC m=+92.048729776" watchObservedRunningTime="2026-07-23 15:01:44.455402462 +0000 UTC m=+92.048853383" Jul 23 15:01:44.483419 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:44.482650 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-dgg29" podStartSLOduration=3.04101188 podStartE2EDuration="25.482629621s" totalImagesPullingTime="22.441617741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:44.481674612 +0000 UTC m=+92.075125522" watchObservedRunningTime="2026-07-23 15:01:44.482629621 +0000 UTC m=+92.076080541" Jul 23 15:01:45.441345 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:45.441308 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" event={"ID":"3d4deab9-b559-49d6-a1eb-d21d6f9111bd","Type":"ContainerStarted","Data":"0f3334b7847b1ee54fdb1eafa7ec3babc69facf2f5f6c7c1584df0076efdfebd"} Jul 23 15:01:45.441886 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:45.441864 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:45.447621 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:45.447590 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" Jul 23 15:01:45.459460 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:45.459408 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-pcbm2" podStartSLOduration=2.752034377 podStartE2EDuration="13.459395058s" totalImagesPullingTime="10.707360681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:45.458362845 +0000 UTC m=+93.051813765" watchObservedRunningTime="2026-07-23 15:01:45.459395058 +0000 UTC m=+93.052845977" Jul 23 15:01:53.472177 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.472147 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jqrct"] Jul 23 15:01:53.533708 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.533680 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9wq85"] Jul 23 15:01:53.533855 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.533801 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.538009 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.537967 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wggg8\"" Jul 23 15:01:53.538009 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.537967 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 15:01:53.538208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.538060 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 15:01:53.538208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.538087 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 15:01:53.538208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.538199 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 15:01:53.538363 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.538295 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 15:01:53.538363 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.538295 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 15:01:53.550702 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.550678 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jqrct"] Jul 23 15:01:53.550819 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.550798 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.553356 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.553337 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 15:01:53.553453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.553410 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 15:01:53.553453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.553423 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 15:01:53.553537 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.553410 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wskx6\"" Jul 23 15:01:53.555776 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.555742 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.555892 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.555793 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea18f270-2af0-4da5-aba8-bdb323a89d6f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.555892 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.555824 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea18f270-2af0-4da5-aba8-bdb323a89d6f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.555892 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.555861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.555892 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.555888 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.556127 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.555933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwr9w\" (UniqueName: \"kubernetes.io/projected/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-api-access-rwr9w\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.656538 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea18f270-2af0-4da5-aba8-bdb323a89d6f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.656684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656545 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.656684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea18f270-2af0-4da5-aba8-bdb323a89d6f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.656684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656583 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.656684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656600 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-sys\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.656684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656645 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.656684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.657023 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656721 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664f3dfa-35fe-4428-99ee-da94508b1d05-metrics-client-ca\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.657023 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656742 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwr9w\" (UniqueName: \"kubernetes.io/projected/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-api-access-rwr9w\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.657023 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:53.656759 2568 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 23 15:01:53.657023 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-root\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.657023 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:01:53.656838 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-tls podName:ea18f270-2af0-4da5-aba8-bdb323a89d6f nodeName:}" failed. No retries permitted until 2026-07-23 15:01:54.156816262 +0000 UTC m=+101.750267161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-jqrct" (UID: "ea18f270-2af0-4da5-aba8-bdb323a89d6f") : secret "kube-state-metrics-tls" not found Jul 23 15:01:53.657023 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.656979 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ea18f270-2af0-4da5-aba8-bdb323a89d6f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.657405 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.657039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p52x9\" (UniqueName: \"kubernetes.io/projected/664f3dfa-35fe-4428-99ee-da94508b1d05-kube-api-access-p52x9\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.657405 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.657076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-wtmp\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.657405 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.657115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-textfile\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.657405 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.657148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-tls\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.657405 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.657189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.657405 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.657374 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea18f270-2af0-4da5-aba8-bdb323a89d6f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.657661 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.657645 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.659057 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.659041 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.665908 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.665883 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwr9w\" (UniqueName: \"kubernetes.io/projected/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-api-access-rwr9w\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:53.758130 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758079 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664f3dfa-35fe-4428-99ee-da94508b1d05-metrics-client-ca\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758130 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-root\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758130 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758128 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p52x9\" (UniqueName: \"kubernetes.io/projected/664f3dfa-35fe-4428-99ee-da94508b1d05-kube-api-access-p52x9\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-wtmp\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758161 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-textfile\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-tls\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758290 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-root\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758301 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-sys\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758582 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758339 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-sys\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758582 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758460 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-wtmp\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758679 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758588 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-textfile\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758679 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758662 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664f3dfa-35fe-4428-99ee-da94508b1d05-metrics-client-ca\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.758841 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.758821 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-accelerators-collector-config\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.760317 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.760296 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-tls\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.760473 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.760453 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664f3dfa-35fe-4428-99ee-da94508b1d05-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.773157 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.773130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p52x9\" (UniqueName: \"kubernetes.io/projected/664f3dfa-35fe-4428-99ee-da94508b1d05-kube-api-access-p52x9\") pod \"node-exporter-9wq85\" (UID: \"664f3dfa-35fe-4428-99ee-da94508b1d05\") " pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.859039 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:53.859016 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9wq85" Jul 23 15:01:53.893926 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:53.893897 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664f3dfa_35fe_4428_99ee_da94508b1d05.slice/crio-60eb826e78a786f7f187a7778878c2dc6de29461a2b2b4abfb1c1d027f0aa5e9 WatchSource:0}: Error finding container 60eb826e78a786f7f187a7778878c2dc6de29461a2b2b4abfb1c1d027f0aa5e9: Status 404 returned error can't find the container with id 60eb826e78a786f7f187a7778878c2dc6de29461a2b2b4abfb1c1d027f0aa5e9 Jul 23 15:01:54.160634 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:54.160560 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:54.162874 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:54.162854 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea18f270-2af0-4da5-aba8-bdb323a89d6f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-jqrct\" (UID: \"ea18f270-2af0-4da5-aba8-bdb323a89d6f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:54.320563 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:54.320536 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6cr64" Jul 23 15:01:54.443229 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:54.443159 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" Jul 23 15:01:54.477045 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:54.477020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wq85" event={"ID":"664f3dfa-35fe-4428-99ee-da94508b1d05","Type":"ContainerStarted","Data":"60eb826e78a786f7f187a7778878c2dc6de29461a2b2b4abfb1c1d027f0aa5e9"} Jul 23 15:01:54.600060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:54.600036 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-jqrct"] Jul 23 15:01:54.601882 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:54.601860 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea18f270_2af0_4da5_aba8_bdb323a89d6f.slice/crio-8d3e30f0e06aafd052ff576bfacf93f950d6eca7ece190f9e37f999a33fb483b WatchSource:0}: Error finding container 8d3e30f0e06aafd052ff576bfacf93f950d6eca7ece190f9e37f999a33fb483b: Status 404 returned error can't find the container with id 8d3e30f0e06aafd052ff576bfacf93f950d6eca7ece190f9e37f999a33fb483b Jul 23 15:01:55.481252 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.481219 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" event={"ID":"ea18f270-2af0-4da5-aba8-bdb323a89d6f","Type":"ContainerStarted","Data":"8d3e30f0e06aafd052ff576bfacf93f950d6eca7ece190f9e37f999a33fb483b"} Jul 23 15:01:55.561839 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.561810 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-766b9c6d48-sjchg"] Jul 23 15:01:55.581315 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.581294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.584491 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.584465 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 23 15:01:55.584733 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.584492 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 23 15:01:55.584733 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.584634 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-x9m7h\"" Jul 23 15:01:55.584733 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.584667 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 23 15:01:55.584903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.584734 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 23 15:01:55.584903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.584876 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 23 15:01:55.585040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.584966 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7hds7sndmstsh\"" Jul 23 15:01:55.592572 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.592548 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-766b9c6d48-sjchg"] Jul 23 15:01:55.672150 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672120 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.672274 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-tls\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.672274 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672237 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-metrics-client-ca\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.672274 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-grpc-tls\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.672453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.672453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672323 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngwc\" (UniqueName: \"kubernetes.io/projected/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-kube-api-access-pngwc\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.672453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672393 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.672453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.672428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773653 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-grpc-tls\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773653 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773618 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773653 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngwc\" (UniqueName: \"kubernetes.io/projected/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-kube-api-access-pngwc\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773935 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773664 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773935 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773689 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773935 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773736 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773935 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773798 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-tls\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.773935 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.773831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-metrics-client-ca\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.774690 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.774604 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-metrics-client-ca\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.776815 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.776787 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-grpc-tls\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.776903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.776810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.776903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.776870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.778349 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.778311 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-tls\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.778631 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.778606 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.778872 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.778853 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.784930 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.784907 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngwc\" (UniqueName: \"kubernetes.io/projected/fc849a00-f7c0-49b5-81aa-1fda9417c5ba-kube-api-access-pngwc\") pod \"thanos-querier-766b9c6d48-sjchg\" (UID: \"fc849a00-f7c0-49b5-81aa-1fda9417c5ba\") " pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:55.894793 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:55.894764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:01:56.040848 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:56.040812 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-766b9c6d48-sjchg"] Jul 23 15:01:56.043271 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:56.043242 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc849a00_f7c0_49b5_81aa_1fda9417c5ba.slice/crio-20a8b27240ce935d6900eaae7e149bf797954e3be309e95b552a53409480f3af WatchSource:0}: Error finding container 20a8b27240ce935d6900eaae7e149bf797954e3be309e95b552a53409480f3af: Status 404 returned error can't find the container with id 20a8b27240ce935d6900eaae7e149bf797954e3be309e95b552a53409480f3af Jul 23 15:01:56.486359 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:56.486320 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" event={"ID":"fc849a00-f7c0-49b5-81aa-1fda9417c5ba","Type":"ContainerStarted","Data":"20a8b27240ce935d6900eaae7e149bf797954e3be309e95b552a53409480f3af"} Jul 23 15:01:56.487789 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:56.487763 2568 generic.go:356] "Generic (PLEG): container finished" podID="664f3dfa-35fe-4428-99ee-da94508b1d05" containerID="ffa5f4ce34bc2f864463f997c08fa1621b5cead598e80b7de2b7da0a28f02ab9" exitCode=0 Jul 23 15:01:56.487915 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:56.487811 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wq85" event={"ID":"664f3dfa-35fe-4428-99ee-da94508b1d05","Type":"ContainerDied","Data":"ffa5f4ce34bc2f864463f997c08fa1621b5cead598e80b7de2b7da0a28f02ab9"} Jul 23 15:01:57.495512 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.495478 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wq85" event={"ID":"664f3dfa-35fe-4428-99ee-da94508b1d05","Type":"ContainerStarted","Data":"f65ab064f52b90bed9c8e8245ac90203b6b7799b92a8f2e6c0304b0b1059a731"} Jul 23 15:01:57.496015 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.495538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9wq85" event={"ID":"664f3dfa-35fe-4428-99ee-da94508b1d05","Type":"ContainerStarted","Data":"d7a029ee985a89446eb3bc2018199045405c2002d9c6f2f8c1e27c009d050eba"} Jul 23 15:01:57.498208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.498179 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" event={"ID":"ea18f270-2af0-4da5-aba8-bdb323a89d6f","Type":"ContainerStarted","Data":"1c8c6703d34bf1391231618621f51f87b8ecc203656ed76680b48d1b10c14474"} Jul 23 15:01:57.498333 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.498233 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" event={"ID":"ea18f270-2af0-4da5-aba8-bdb323a89d6f","Type":"ContainerStarted","Data":"7b642a35fa840af27c761cf8e10f2ca08cd17f44a5e30413395d1d797c59f91b"} Jul 23 15:01:57.498333 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.498263 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" event={"ID":"ea18f270-2af0-4da5-aba8-bdb323a89d6f","Type":"ContainerStarted","Data":"75c21644b597fe34bf6a7c0a49fcd7cd199d9acf95e521bb10605d155359d193"} Jul 23 15:01:57.523097 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.523056 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9wq85" podStartSLOduration=2.719075634 podStartE2EDuration="4.523042839s" totalImagesPullingTime="1.711148066s" totalInitContainerRuntime="92.819139ms" isStatefulPod=true podCreationTimestamp="2026-07-23 15:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:57.521920504 +0000 UTC m=+105.115371424" watchObservedRunningTime="2026-07-23 15:01:57.523042839 +0000 UTC m=+105.116493758" Jul 23 15:01:57.545593 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.545516 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-jqrct" podStartSLOduration=2.18731013 podStartE2EDuration="4.545502051s" totalImagesPullingTime="2.358191921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:01:57.544217845 +0000 UTC m=+105.137668767" watchObservedRunningTime="2026-07-23 15:01:57.545502051 +0000 UTC m=+105.138952970" Jul 23 15:01:57.872019 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.871926 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-876bc649b-f8bq8"] Jul 23 15:01:57.891237 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.891211 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-876bc649b-f8bq8"] Jul 23 15:01:57.891352 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.891313 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:57.895504 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.895477 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-22k28n6j01eko\"" Jul 23 15:01:57.896422 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.896406 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 23 15:01:57.896517 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.896490 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 23 15:01:57.896517 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.896507 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 23 15:01:57.896684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.896603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 23 15:01:57.896744 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.896734 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zvkc4\"" Jul 23 15:01:57.992659 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.992634 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-secret-metrics-server-tls\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:57.992784 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.992684 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28gs8\" (UniqueName: \"kubernetes.io/projected/3b9f08ca-1022-45fd-8943-e549ba656172-kube-api-access-28gs8\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:57.992784 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.992728 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-secret-metrics-server-client-certs\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:57.992784 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.992750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3b9f08ca-1022-45fd-8943-e549ba656172-audit-log\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:57.992961 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.992869 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3b9f08ca-1022-45fd-8943-e549ba656172-metrics-server-audit-profiles\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:57.992961 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.992925 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9f08ca-1022-45fd-8943-e549ba656172-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:57.993077 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:57.992980 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-client-ca-bundle\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.093881 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.093857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-secret-metrics-server-tls\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094029 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.093893 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28gs8\" (UniqueName: \"kubernetes.io/projected/3b9f08ca-1022-45fd-8943-e549ba656172-kube-api-access-28gs8\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094029 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.093911 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-secret-metrics-server-client-certs\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094029 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.093927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3b9f08ca-1022-45fd-8943-e549ba656172-audit-log\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094029 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.093943 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3b9f08ca-1022-45fd-8943-e549ba656172-metrics-server-audit-profiles\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094029 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.093966 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9f08ca-1022-45fd-8943-e549ba656172-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094302 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.094104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-client-ca-bundle\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094398 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.094378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3b9f08ca-1022-45fd-8943-e549ba656172-audit-log\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094773 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.094745 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9f08ca-1022-45fd-8943-e549ba656172-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.094986 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.094968 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3b9f08ca-1022-45fd-8943-e549ba656172-metrics-server-audit-profiles\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.096259 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.096236 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-secret-metrics-server-tls\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.096363 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.096318 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-client-ca-bundle\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.096479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.096460 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3b9f08ca-1022-45fd-8943-e549ba656172-secret-metrics-server-client-certs\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.102975 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.102958 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28gs8\" (UniqueName: \"kubernetes.io/projected/3b9f08ca-1022-45fd-8943-e549ba656172-kube-api-access-28gs8\") pod \"metrics-server-876bc649b-f8bq8\" (UID: \"3b9f08ca-1022-45fd-8943-e549ba656172\") " pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.201540 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.201506 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:01:58.812760 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:58.812738 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-876bc649b-f8bq8"] Jul 23 15:01:58.814629 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:01:58.814600 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9f08ca_1022_45fd_8943_e549ba656172.slice/crio-83369f33fd4c38f957615372b8ee8a6a33fe762be2525d4a9422501e97bc77ee WatchSource:0}: Error finding container 83369f33fd4c38f957615372b8ee8a6a33fe762be2525d4a9422501e97bc77ee: Status 404 returned error can't find the container with id 83369f33fd4c38f957615372b8ee8a6a33fe762be2525d4a9422501e97bc77ee Jul 23 15:01:59.508850 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:59.508808 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" event={"ID":"fc849a00-f7c0-49b5-81aa-1fda9417c5ba","Type":"ContainerStarted","Data":"7e0a6f422f41b25fbaf9a85d243d287c5b280b6bd728ece09e166da75499db3b"} Jul 23 15:01:59.509050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:59.508898 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" event={"ID":"fc849a00-f7c0-49b5-81aa-1fda9417c5ba","Type":"ContainerStarted","Data":"0dc232281516fc8a2b5489e40bfd63a6a79138c432bc01472897e8203b8e19b6"} Jul 23 15:01:59.509050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:59.508939 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" event={"ID":"fc849a00-f7c0-49b5-81aa-1fda9417c5ba","Type":"ContainerStarted","Data":"8d422aec823219b41d3d7038611cabf510b717dda3879c46e6afecc90daeeae0"} Jul 23 15:01:59.510267 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:01:59.510238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" event={"ID":"3b9f08ca-1022-45fd-8943-e549ba656172","Type":"ContainerStarted","Data":"83369f33fd4c38f957615372b8ee8a6a33fe762be2525d4a9422501e97bc77ee"} Jul 23 15:02:01.522743 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:01.522706 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" event={"ID":"3b9f08ca-1022-45fd-8943-e549ba656172","Type":"ContainerStarted","Data":"74afbbea2d378fba417a8c359c61b4435d054fca5288231a53ae7b41f7a12b9c"} Jul 23 15:02:01.541848 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:01.541801 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" podStartSLOduration=2.5602302139999997 podStartE2EDuration="4.54178627s" totalImagesPullingTime="1.981556056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:02:01.541101478 +0000 UTC m=+109.134552398" watchObservedRunningTime="2026-07-23 15:02:01.54178627 +0000 UTC m=+109.135237188" Jul 23 15:02:02.529164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:02.529120 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" event={"ID":"fc849a00-f7c0-49b5-81aa-1fda9417c5ba","Type":"ContainerStarted","Data":"a865f94565fe037b83c16e591af4d0deaa019fba42a83efcb64c10ddc7b0a2a0"} Jul 23 15:02:02.529523 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:02.529208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" event={"ID":"fc849a00-f7c0-49b5-81aa-1fda9417c5ba","Type":"ContainerStarted","Data":"916a5eb9d5faef336cfdc6825df0cb75bcfce40aafa289a44533732c91a74e01"} Jul 23 15:02:02.529523 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:02.529252 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" event={"ID":"fc849a00-f7c0-49b5-81aa-1fda9417c5ba","Type":"ContainerStarted","Data":"ca8132918fe4db2c904438193f4d86f14b46a2dc685132ac10a403e8598a2e76"} Jul 23 15:02:02.555247 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:02.555193 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" podStartSLOduration=2.434823907 podStartE2EDuration="7.555174057s" totalImagesPullingTime="5.12035015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:01:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:02:02.553336582 +0000 UTC m=+110.146787531" watchObservedRunningTime="2026-07-23 15:02:02.555174057 +0000 UTC m=+110.148624978" Jul 23 15:02:03.533307 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:03.533277 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:02:04.541592 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:04.541566 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-766b9c6d48-sjchg" Jul 23 15:02:08.437957 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.437895 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" podUID="a7a2dd24-8428-4072-8938-eeeb2895ecc5" containerName="registry" containerID="cri-o://e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5" gracePeriod=30 Jul 23 15:02:08.702640 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.702612 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:02:08.772806 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.772781 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-bound-sa-token\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.772982 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.772821 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-image-registry-private-configuration\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.772982 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.772850 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-trusted-ca\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.772982 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.772867 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs9n5\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-kube-api-access-qs9n5\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.773183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.773022 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-installation-pull-secrets\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.773183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.773066 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.773183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.773091 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-certificates\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.773183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.773172 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7a2dd24-8428-4072-8938-eeeb2895ecc5-ca-trust-extracted\") pod \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\" (UID: \"a7a2dd24-8428-4072-8938-eeeb2895ecc5\") " Jul 23 15:02:08.773387 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.773357 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 15:02:08.773589 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.773475 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-trusted-ca\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:08.773723 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.773697 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 15:02:08.775473 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.775424 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:02:08.775587 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.775565 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:02:08.775684 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.775665 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-kube-api-access-qs9n5" (OuterVolumeSpecName: "kube-api-access-qs9n5") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "kube-api-access-qs9n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:02:08.775739 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.775667 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:02:08.775739 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.775724 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:02:08.781936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.781913 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7a2dd24-8428-4072-8938-eeeb2895ecc5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a7a2dd24-8428-4072-8938-eeeb2895ecc5" (UID: "a7a2dd24-8428-4072-8938-eeeb2895ecc5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:02:08.873939 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.873896 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-installation-pull-secrets\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:08.873939 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.873932 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-tls\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:08.873939 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.873944 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7a2dd24-8428-4072-8938-eeeb2895ecc5-registry-certificates\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:08.874202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.873956 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7a2dd24-8428-4072-8938-eeeb2895ecc5-ca-trust-extracted\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:08.874202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.873968 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-bound-sa-token\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:08.874202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.873980 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7a2dd24-8428-4072-8938-eeeb2895ecc5-image-registry-private-configuration\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:08.874202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:08.874021 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9n5\" (UniqueName: \"kubernetes.io/projected/a7a2dd24-8428-4072-8938-eeeb2895ecc5-kube-api-access-qs9n5\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:02:09.554072 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.554038 2568 generic.go:356] "Generic (PLEG): container finished" podID="a7a2dd24-8428-4072-8938-eeeb2895ecc5" containerID="e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5" exitCode=0 Jul 23 15:02:09.554508 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.554082 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" event={"ID":"a7a2dd24-8428-4072-8938-eeeb2895ecc5","Type":"ContainerDied","Data":"e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5"} Jul 23 15:02:09.554508 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.554091 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" Jul 23 15:02:09.554508 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.554142 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bcd859484-9xr7k" event={"ID":"a7a2dd24-8428-4072-8938-eeeb2895ecc5","Type":"ContainerDied","Data":"4fa549ac6d085eba80a3032bf261b41417af7fe7831c99c11bb264c541e3deca"} Jul 23 15:02:09.554508 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.554163 2568 scope.go:117] "RemoveContainer" containerID="e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5" Jul 23 15:02:09.562301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.562282 2568 scope.go:117] "RemoveContainer" containerID="e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5" Jul 23 15:02:09.562574 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:02:09.562543 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5\": container with ID starting with e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5 not found: ID does not exist" containerID="e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5" Jul 23 15:02:09.562673 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.562572 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5"} err="failed to get container status \"e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5\": rpc error: code = NotFound desc = could not find container \"e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5\": container with ID starting with e4eac7c355f043c0b06fef7d8dc763092a9d1030ba174ee665a89f8bae0c18f5 not found: ID does not exist" Jul 23 15:02:09.572970 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.572951 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bcd859484-9xr7k"] Jul 23 15:02:09.579458 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:09.579439 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bcd859484-9xr7k"] Jul 23 15:02:10.963575 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:10.963543 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a2dd24-8428-4072-8938-eeeb2895ecc5" path="/var/lib/kubelet/pods/a7a2dd24-8428-4072-8938-eeeb2895ecc5/volumes" Jul 23 15:02:14.575349 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:14.575318 2568 generic.go:356] "Generic (PLEG): container finished" podID="7924e63c-f6ec-4008-bc30-2d35fc835b7c" containerID="35ec9f84f91f7e90deb86916dda18243b243d1e1b87151415e6b43c13333d0ae" exitCode=0 Jul 23 15:02:14.575701 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:14.575381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" event={"ID":"7924e63c-f6ec-4008-bc30-2d35fc835b7c","Type":"ContainerDied","Data":"35ec9f84f91f7e90deb86916dda18243b243d1e1b87151415e6b43c13333d0ae"} Jul 23 15:02:14.575701 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:14.575633 2568 scope.go:117] "RemoveContainer" containerID="35ec9f84f91f7e90deb86916dda18243b243d1e1b87151415e6b43c13333d0ae" Jul 23 15:02:15.580727 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:15.580694 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-69l6l" event={"ID":"7924e63c-f6ec-4008-bc30-2d35fc835b7c","Type":"ContainerStarted","Data":"9194fc2e3ccf5093bb8660597bee1134b22091a93f607b9525b6e1771864543b"} Jul 23 15:02:18.202699 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:18.202664 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:02:18.203122 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:18.202752 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:02:29.638730 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:29.638697 2568 generic.go:356] "Generic (PLEG): container finished" podID="a45bae56-0ce9-4902-ba2a-604c879907cd" containerID="a0bb54fc9c3313b8a15a4cc2e75099c5ff2eef4ed466a84fd1c7b7768f243e09" exitCode=0 Jul 23 15:02:29.639164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:29.638771 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-lf7g5" event={"ID":"a45bae56-0ce9-4902-ba2a-604c879907cd","Type":"ContainerDied","Data":"a0bb54fc9c3313b8a15a4cc2e75099c5ff2eef4ed466a84fd1c7b7768f243e09"} Jul 23 15:02:29.639164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:29.639094 2568 scope.go:117] "RemoveContainer" containerID="a0bb54fc9c3313b8a15a4cc2e75099c5ff2eef4ed466a84fd1c7b7768f243e09" Jul 23 15:02:30.643899 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:30.643864 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-lf7g5" event={"ID":"a45bae56-0ce9-4902-ba2a-604c879907cd","Type":"ContainerStarted","Data":"302e5049cd0c4c26fac8aa8b848aad8e1d6d6a161554d634d77cd240f57ca905"} Jul 23 15:02:38.206922 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:38.206889 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:02:38.210612 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:02:38.210586 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-876bc649b-f8bq8" Jul 23 15:05:12.847060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:12.847024 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:05:12.847614 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:12.847432 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:05:12.851618 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:12.851597 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:05:12.851751 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:12.851600 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:05:12.858354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:12.858337 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 15:05:43.198660 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.198624 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj"] Jul 23 15:05:43.201641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.200131 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a2dd24-8428-4072-8938-eeeb2895ecc5" containerName="registry" Jul 23 15:05:43.201641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.200156 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a2dd24-8428-4072-8938-eeeb2895ecc5" containerName="registry" Jul 23 15:05:43.201641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.200327 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a2dd24-8428-4072-8938-eeeb2895ecc5" containerName="registry" Jul 23 15:05:43.204668 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.204638 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.207614 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.207590 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 15:05:43.207614 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.207604 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 15:05:43.207785 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.207604 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v29lk\"" Jul 23 15:05:43.213189 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.213165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj"] Jul 23 15:05:43.285369 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.285341 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvc9\" (UniqueName: \"kubernetes.io/projected/29be97ae-a547-4eef-b693-29d9c0d413b6-kube-api-access-mlvc9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.285476 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.285403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.285476 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.285432 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.386095 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.386073 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvc9\" (UniqueName: \"kubernetes.io/projected/29be97ae-a547-4eef-b693-29d9c0d413b6-kube-api-access-mlvc9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.386195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.386120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.386195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.386137 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.386539 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.386520 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.386577 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.386535 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.397755 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.397729 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvc9\" (UniqueName: \"kubernetes.io/projected/29be97ae-a547-4eef-b693-29d9c0d413b6-kube-api-access-mlvc9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.514180 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.514130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:05:43.642809 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.642783 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj"] Jul 23 15:05:43.644567 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:05:43.644544 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29be97ae_a547_4eef_b693_29d9c0d413b6.slice/crio-c74050216ecd88f5b5a901d229935d1e44ccae94bede0c6e1622abaa953a7132 WatchSource:0}: Error finding container c74050216ecd88f5b5a901d229935d1e44ccae94bede0c6e1622abaa953a7132: Status 404 returned error can't find the container with id c74050216ecd88f5b5a901d229935d1e44ccae94bede0c6e1622abaa953a7132 Jul 23 15:05:43.646420 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:43.646405 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 15:05:44.381542 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:44.381511 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" event={"ID":"29be97ae-a547-4eef-b693-29d9c0d413b6","Type":"ContainerStarted","Data":"c74050216ecd88f5b5a901d229935d1e44ccae94bede0c6e1622abaa953a7132"} Jul 23 15:05:50.408670 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:50.408576 2568 generic.go:356] "Generic (PLEG): container finished" podID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerID="db77d6ce28909c9c00b2ce91c0c2c00d9b7cdeea7a81160fa7ac120647d508ba" exitCode=0 Jul 23 15:05:50.409050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:50.408663 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" event={"ID":"29be97ae-a547-4eef-b693-29d9c0d413b6","Type":"ContainerDied","Data":"db77d6ce28909c9c00b2ce91c0c2c00d9b7cdeea7a81160fa7ac120647d508ba"} Jul 23 15:05:52.418562 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:52.418533 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" event={"ID":"29be97ae-a547-4eef-b693-29d9c0d413b6","Type":"ContainerStarted","Data":"90f651df0b485a1006f22e41e9fc36a0e6a6961bcb82a8179c0027f25687ad1a"} Jul 23 15:05:53.423753 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:53.423724 2568 generic.go:356] "Generic (PLEG): container finished" podID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerID="90f651df0b485a1006f22e41e9fc36a0e6a6961bcb82a8179c0027f25687ad1a" exitCode=0 Jul 23 15:05:53.424183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:05:53.423775 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" event={"ID":"29be97ae-a547-4eef-b693-29d9c0d413b6","Type":"ContainerDied","Data":"90f651df0b485a1006f22e41e9fc36a0e6a6961bcb82a8179c0027f25687ad1a"} Jul 23 15:06:00.456517 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:00.456478 2568 generic.go:356] "Generic (PLEG): container finished" podID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerID="6f70a59a9148ac25f545f7ba92f2a05515be829a1074ad2f2d2d0a4c6f627004" exitCode=0 Jul 23 15:06:00.456897 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:00.456534 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" event={"ID":"29be97ae-a547-4eef-b693-29d9c0d413b6","Type":"ContainerDied","Data":"6f70a59a9148ac25f545f7ba92f2a05515be829a1074ad2f2d2d0a4c6f627004"} Jul 23 15:06:01.585111 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.585088 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:06:01.639022 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.638979 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlvc9\" (UniqueName: \"kubernetes.io/projected/29be97ae-a547-4eef-b693-29d9c0d413b6-kube-api-access-mlvc9\") pod \"29be97ae-a547-4eef-b693-29d9c0d413b6\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " Jul 23 15:06:01.639143 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.639026 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-util\") pod \"29be97ae-a547-4eef-b693-29d9c0d413b6\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " Jul 23 15:06:01.639143 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.639059 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-bundle\") pod \"29be97ae-a547-4eef-b693-29d9c0d413b6\" (UID: \"29be97ae-a547-4eef-b693-29d9c0d413b6\") " Jul 23 15:06:01.639678 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.639656 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-bundle" (OuterVolumeSpecName: "bundle") pod "29be97ae-a547-4eef-b693-29d9c0d413b6" (UID: "29be97ae-a547-4eef-b693-29d9c0d413b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:06:01.641058 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.641021 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29be97ae-a547-4eef-b693-29d9c0d413b6-kube-api-access-mlvc9" (OuterVolumeSpecName: "kube-api-access-mlvc9") pod "29be97ae-a547-4eef-b693-29d9c0d413b6" (UID: "29be97ae-a547-4eef-b693-29d9c0d413b6"). InnerVolumeSpecName "kube-api-access-mlvc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:06:01.643936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.643918 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-util" (OuterVolumeSpecName: "util") pod "29be97ae-a547-4eef-b693-29d9c0d413b6" (UID: "29be97ae-a547-4eef-b693-29d9c0d413b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:06:01.739696 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.739643 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvc9\" (UniqueName: \"kubernetes.io/projected/29be97ae-a547-4eef-b693-29d9c0d413b6-kube-api-access-mlvc9\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:01.739696 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.739666 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-util\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:01.739696 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:01.739676 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/29be97ae-a547-4eef-b693-29d9c0d413b6-bundle\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:02.466341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:02.466314 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" Jul 23 15:06:02.466341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:02.466334 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dt9kj" event={"ID":"29be97ae-a547-4eef-b693-29d9c0d413b6","Type":"ContainerDied","Data":"c74050216ecd88f5b5a901d229935d1e44ccae94bede0c6e1622abaa953a7132"} Jul 23 15:06:02.466578 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:02.466383 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c74050216ecd88f5b5a901d229935d1e44ccae94bede0c6e1622abaa953a7132" Jul 23 15:06:06.494971 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.494895 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb"] Jul 23 15:06:06.495341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.495217 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerName="extract" Jul 23 15:06:06.495341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.495228 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerName="extract" Jul 23 15:06:06.495341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.495236 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerName="pull" Jul 23 15:06:06.495341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.495242 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerName="pull" Jul 23 15:06:06.495341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.495258 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerName="util" Jul 23 15:06:06.495341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.495264 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerName="util" Jul 23 15:06:06.495341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.495308 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="29be97ae-a547-4eef-b693-29d9c0d413b6" containerName="extract" Jul 23 15:06:06.498188 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.498171 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.501077 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.501050 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 23 15:06:06.501253 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.501225 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 23 15:06:06.501333 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.501313 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-tnbrh\"" Jul 23 15:06:06.524613 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.524585 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb"] Jul 23 15:06:06.573116 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.573092 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c44f34c2-63e7-479a-af5d-7e9562e2cb19-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kz4kb\" (UID: \"c44f34c2-63e7-479a-af5d-7e9562e2cb19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.573211 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.573140 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7xl\" (UniqueName: \"kubernetes.io/projected/c44f34c2-63e7-479a-af5d-7e9562e2cb19-kube-api-access-9q7xl\") pod \"cert-manager-operator-controller-manager-55c444bf86-kz4kb\" (UID: \"c44f34c2-63e7-479a-af5d-7e9562e2cb19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.673909 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.673875 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7xl\" (UniqueName: \"kubernetes.io/projected/c44f34c2-63e7-479a-af5d-7e9562e2cb19-kube-api-access-9q7xl\") pod \"cert-manager-operator-controller-manager-55c444bf86-kz4kb\" (UID: \"c44f34c2-63e7-479a-af5d-7e9562e2cb19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.674029 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.673950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c44f34c2-63e7-479a-af5d-7e9562e2cb19-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kz4kb\" (UID: \"c44f34c2-63e7-479a-af5d-7e9562e2cb19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.674299 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.674280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c44f34c2-63e7-479a-af5d-7e9562e2cb19-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kz4kb\" (UID: \"c44f34c2-63e7-479a-af5d-7e9562e2cb19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.684084 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.684063 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7xl\" (UniqueName: \"kubernetes.io/projected/c44f34c2-63e7-479a-af5d-7e9562e2cb19-kube-api-access-9q7xl\") pod \"cert-manager-operator-controller-manager-55c444bf86-kz4kb\" (UID: \"c44f34c2-63e7-479a-af5d-7e9562e2cb19\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.806694 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.806642 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" Jul 23 15:06:06.941661 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:06.941641 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb"] Jul 23 15:06:06.943617 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:06:06.943592 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44f34c2_63e7_479a_af5d_7e9562e2cb19.slice/crio-bf60a2b299a038438c0ab068322b603e612f5eb59e18e0aaab23c73df4c78813 WatchSource:0}: Error finding container bf60a2b299a038438c0ab068322b603e612f5eb59e18e0aaab23c73df4c78813: Status 404 returned error can't find the container with id bf60a2b299a038438c0ab068322b603e612f5eb59e18e0aaab23c73df4c78813 Jul 23 15:06:07.487861 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:07.487833 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" event={"ID":"c44f34c2-63e7-479a-af5d-7e9562e2cb19","Type":"ContainerStarted","Data":"bf60a2b299a038438c0ab068322b603e612f5eb59e18e0aaab23c73df4c78813"} Jul 23 15:06:09.498492 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:09.498454 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" event={"ID":"c44f34c2-63e7-479a-af5d-7e9562e2cb19","Type":"ContainerStarted","Data":"fbf523f2b425b9e8c73ebe31590811df0500deb91f7bd94054226c67dbb9f270"} Jul 23 15:06:09.532907 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:09.532862 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kz4kb" podStartSLOduration=1.8742627920000001 podStartE2EDuration="3.532848496s" totalImagesPullingTime="1.658585704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:06:09.531883989 +0000 UTC m=+357.125334933" watchObservedRunningTime="2026-07-23 15:06:09.532848496 +0000 UTC m=+357.126299459" Jul 23 15:06:10.176942 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.176908 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw"] Jul 23 15:06:10.180399 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.180380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.191105 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.191084 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 15:06:10.191190 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.191138 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v29lk\"" Jul 23 15:06:10.191314 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.191300 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 15:06:10.204748 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.204727 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw"] Jul 23 15:06:10.303864 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.303829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.304040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.303872 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.304040 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.303902 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj4f8\" (UniqueName: \"kubernetes.io/projected/08b6f75f-c882-4811-b287-370409df0283-kube-api-access-gj4f8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.404688 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.404657 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.404838 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.404697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.404838 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.404717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj4f8\" (UniqueName: \"kubernetes.io/projected/08b6f75f-c882-4811-b287-370409df0283-kube-api-access-gj4f8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.405156 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.405135 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.405156 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.405147 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.424145 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.424120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj4f8\" (UniqueName: \"kubernetes.io/projected/08b6f75f-c882-4811-b287-370409df0283-kube-api-access-gj4f8\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.489029 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.489007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:10.627656 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:10.627626 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw"] Jul 23 15:06:10.630643 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:06:10.630610 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b6f75f_c882_4811_b287_370409df0283.slice/crio-317c97235652c6bf44b0adbe1e8937e0e5486aaf2fc562145ed655daefefb933 WatchSource:0}: Error finding container 317c97235652c6bf44b0adbe1e8937e0e5486aaf2fc562145ed655daefefb933: Status 404 returned error can't find the container with id 317c97235652c6bf44b0adbe1e8937e0e5486aaf2fc562145ed655daefefb933 Jul 23 15:06:11.509519 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:11.509489 2568 generic.go:356] "Generic (PLEG): container finished" podID="08b6f75f-c882-4811-b287-370409df0283" containerID="7932fbf85ca0571c43446b8f2792782903e228a6f13555997b378c280ca2795b" exitCode=0 Jul 23 15:06:11.509642 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:11.509517 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" event={"ID":"08b6f75f-c882-4811-b287-370409df0283","Type":"ContainerDied","Data":"7932fbf85ca0571c43446b8f2792782903e228a6f13555997b378c280ca2795b"} Jul 23 15:06:11.509642 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:11.509592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" event={"ID":"08b6f75f-c882-4811-b287-370409df0283","Type":"ContainerStarted","Data":"317c97235652c6bf44b0adbe1e8937e0e5486aaf2fc562145ed655daefefb933"} Jul 23 15:06:13.502454 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.502416 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2"] Jul 23 15:06:13.514395 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.514353 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.517356 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.517298 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2"] Jul 23 15:06:13.517498 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.517464 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 23 15:06:13.517724 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.517706 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gg9cv\"" Jul 23 15:06:13.518730 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.518646 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 23 15:06:13.633155 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.633125 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c4673ef-5285-42ee-96ce-927dbd710f8e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvbq2\" (UID: \"4c4673ef-5285-42ee-96ce-927dbd710f8e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.633287 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.633162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvmnc\" (UniqueName: \"kubernetes.io/projected/4c4673ef-5285-42ee-96ce-927dbd710f8e-kube-api-access-dvmnc\") pod \"cert-manager-cainjector-559cdd8c87-mvbq2\" (UID: \"4c4673ef-5285-42ee-96ce-927dbd710f8e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.733671 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.733643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c4673ef-5285-42ee-96ce-927dbd710f8e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvbq2\" (UID: \"4c4673ef-5285-42ee-96ce-927dbd710f8e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.733771 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.733687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvmnc\" (UniqueName: \"kubernetes.io/projected/4c4673ef-5285-42ee-96ce-927dbd710f8e-kube-api-access-dvmnc\") pod \"cert-manager-cainjector-559cdd8c87-mvbq2\" (UID: \"4c4673ef-5285-42ee-96ce-927dbd710f8e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.743873 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.743839 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c4673ef-5285-42ee-96ce-927dbd710f8e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mvbq2\" (UID: \"4c4673ef-5285-42ee-96ce-927dbd710f8e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.743983 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.743924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvmnc\" (UniqueName: \"kubernetes.io/projected/4c4673ef-5285-42ee-96ce-927dbd710f8e-kube-api-access-dvmnc\") pod \"cert-manager-cainjector-559cdd8c87-mvbq2\" (UID: \"4c4673ef-5285-42ee-96ce-927dbd710f8e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.838150 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.838123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" Jul 23 15:06:13.967420 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:13.967396 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2"] Jul 23 15:06:13.969280 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:06:13.969253 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c4673ef_5285_42ee_96ce_927dbd710f8e.slice/crio-d63255cecf596e35fe5a1b59ffd2999870f0ddf796b096aad6300b2d6f3288e4 WatchSource:0}: Error finding container d63255cecf596e35fe5a1b59ffd2999870f0ddf796b096aad6300b2d6f3288e4: Status 404 returned error can't find the container with id d63255cecf596e35fe5a1b59ffd2999870f0ddf796b096aad6300b2d6f3288e4 Jul 23 15:06:14.526273 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:14.526238 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" event={"ID":"4c4673ef-5285-42ee-96ce-927dbd710f8e","Type":"ContainerStarted","Data":"d63255cecf596e35fe5a1b59ffd2999870f0ddf796b096aad6300b2d6f3288e4"} Jul 23 15:06:14.527954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:14.527930 2568 generic.go:356] "Generic (PLEG): container finished" podID="08b6f75f-c882-4811-b287-370409df0283" containerID="56acd591b43175f9e108765b0898338f3ba87990c3bcf2869ad4aadd34178a78" exitCode=0 Jul 23 15:06:14.528097 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:14.528015 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" event={"ID":"08b6f75f-c882-4811-b287-370409df0283","Type":"ContainerDied","Data":"56acd591b43175f9e108765b0898338f3ba87990c3bcf2869ad4aadd34178a78"} Jul 23 15:06:15.535447 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:15.535410 2568 generic.go:356] "Generic (PLEG): container finished" podID="08b6f75f-c882-4811-b287-370409df0283" containerID="51bc33a926e4d32a29d104c2f019b975c58079b04666dd2be1536fa7b6a81fec" exitCode=0 Jul 23 15:06:15.535909 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:15.535488 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" event={"ID":"08b6f75f-c882-4811-b287-370409df0283","Type":"ContainerDied","Data":"51bc33a926e4d32a29d104c2f019b975c58079b04666dd2be1536fa7b6a81fec"} Jul 23 15:06:16.889857 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:16.889836 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:16.960143 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:16.960122 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj4f8\" (UniqueName: \"kubernetes.io/projected/08b6f75f-c882-4811-b287-370409df0283-kube-api-access-gj4f8\") pod \"08b6f75f-c882-4811-b287-370409df0283\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " Jul 23 15:06:16.960230 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:16.960182 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-bundle\") pod \"08b6f75f-c882-4811-b287-370409df0283\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " Jul 23 15:06:16.960230 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:16.960222 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-util\") pod \"08b6f75f-c882-4811-b287-370409df0283\" (UID: \"08b6f75f-c882-4811-b287-370409df0283\") " Jul 23 15:06:16.960588 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:16.960559 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-bundle" (OuterVolumeSpecName: "bundle") pod "08b6f75f-c882-4811-b287-370409df0283" (UID: "08b6f75f-c882-4811-b287-370409df0283"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:06:16.962141 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:16.962113 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b6f75f-c882-4811-b287-370409df0283-kube-api-access-gj4f8" (OuterVolumeSpecName: "kube-api-access-gj4f8") pod "08b6f75f-c882-4811-b287-370409df0283" (UID: "08b6f75f-c882-4811-b287-370409df0283"). InnerVolumeSpecName "kube-api-access-gj4f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:06:16.964550 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:16.964519 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-util" (OuterVolumeSpecName: "util") pod "08b6f75f-c882-4811-b287-370409df0283" (UID: "08b6f75f-c882-4811-b287-370409df0283"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:06:17.060878 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.060851 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj4f8\" (UniqueName: \"kubernetes.io/projected/08b6f75f-c882-4811-b287-370409df0283-kube-api-access-gj4f8\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:17.060878 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.060878 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-bundle\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:17.061086 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.060888 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08b6f75f-c882-4811-b287-370409df0283-util\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:17.547237 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.547199 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" event={"ID":"4c4673ef-5285-42ee-96ce-927dbd710f8e","Type":"ContainerStarted","Data":"3a881744fb36664c9be8377c29af6cceb6c0cdf854ca45c2d4b04165dbdd9426"} Jul 23 15:06:17.549384 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.549354 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" event={"ID":"08b6f75f-c882-4811-b287-370409df0283","Type":"ContainerDied","Data":"317c97235652c6bf44b0adbe1e8937e0e5486aaf2fc562145ed655daefefb933"} Jul 23 15:06:17.549384 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.549368 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzhvtw" Jul 23 15:06:17.549572 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.549398 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317c97235652c6bf44b0adbe1e8937e0e5486aaf2fc562145ed655daefefb933" Jul 23 15:06:17.576732 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:17.576691 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mvbq2" podStartSLOduration=1.612438107 podStartE2EDuration="4.576678072s" totalImagesPullingTime="2.964239965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:06:17.574272832 +0000 UTC m=+365.167723751" watchObservedRunningTime="2026-07-23 15:06:17.576678072 +0000 UTC m=+365.170128990" Jul 23 15:06:22.854235 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854201 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4cz88"] Jul 23 15:06:22.854689 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854639 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b6f75f-c882-4811-b287-370409df0283" containerName="util" Jul 23 15:06:22.854689 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854657 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b6f75f-c882-4811-b287-370409df0283" containerName="util" Jul 23 15:06:22.854689 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854668 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b6f75f-c882-4811-b287-370409df0283" containerName="pull" Jul 23 15:06:22.854689 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854675 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b6f75f-c882-4811-b287-370409df0283" containerName="pull" Jul 23 15:06:22.854689 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854687 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b6f75f-c882-4811-b287-370409df0283" containerName="extract" Jul 23 15:06:22.854934 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854696 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b6f75f-c882-4811-b287-370409df0283" containerName="extract" Jul 23 15:06:22.854934 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.854778 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b6f75f-c882-4811-b287-370409df0283" containerName="extract" Jul 23 15:06:22.859614 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.859594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:22.862529 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.862513 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-k86kc\"" Jul 23 15:06:22.868836 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:22.868815 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4cz88"] Jul 23 15:06:23.004137 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.004112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdtn\" (UniqueName: \"kubernetes.io/projected/73c61954-9500-4f61-aee9-6caa8c9bd5bf-kube-api-access-jzdtn\") pod \"cert-manager-55b4d954d5-4cz88\" (UID: \"73c61954-9500-4f61-aee9-6caa8c9bd5bf\") " pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:23.004263 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.004178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73c61954-9500-4f61-aee9-6caa8c9bd5bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-4cz88\" (UID: \"73c61954-9500-4f61-aee9-6caa8c9bd5bf\") " pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:23.104700 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.104634 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73c61954-9500-4f61-aee9-6caa8c9bd5bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-4cz88\" (UID: \"73c61954-9500-4f61-aee9-6caa8c9bd5bf\") " pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:23.104700 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.104676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdtn\" (UniqueName: \"kubernetes.io/projected/73c61954-9500-4f61-aee9-6caa8c9bd5bf-kube-api-access-jzdtn\") pod \"cert-manager-55b4d954d5-4cz88\" (UID: \"73c61954-9500-4f61-aee9-6caa8c9bd5bf\") " pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:23.115308 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.115287 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73c61954-9500-4f61-aee9-6caa8c9bd5bf-bound-sa-token\") pod \"cert-manager-55b4d954d5-4cz88\" (UID: \"73c61954-9500-4f61-aee9-6caa8c9bd5bf\") " pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:23.115546 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.115527 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdtn\" (UniqueName: \"kubernetes.io/projected/73c61954-9500-4f61-aee9-6caa8c9bd5bf-kube-api-access-jzdtn\") pod \"cert-manager-55b4d954d5-4cz88\" (UID: \"73c61954-9500-4f61-aee9-6caa8c9bd5bf\") " pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:23.169234 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.169214 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4cz88" Jul 23 15:06:23.299656 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.299632 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4cz88"] Jul 23 15:06:23.301486 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:06:23.301459 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73c61954_9500_4f61_aee9_6caa8c9bd5bf.slice/crio-4c882796b32d8ad310a800ce4c4d8f4e088aa27b0b06228e41c3a30c66f3e35b WatchSource:0}: Error finding container 4c882796b32d8ad310a800ce4c4d8f4e088aa27b0b06228e41c3a30c66f3e35b: Status 404 returned error can't find the container with id 4c882796b32d8ad310a800ce4c4d8f4e088aa27b0b06228e41c3a30c66f3e35b Jul 23 15:06:23.576442 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.576408 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4cz88" event={"ID":"73c61954-9500-4f61-aee9-6caa8c9bd5bf","Type":"ContainerStarted","Data":"b21e8bfe4be9d3bfde91bfb680cfba63833d5f149eac49af2a097a81e7febd14"} Jul 23 15:06:23.576442 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.576448 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4cz88" event={"ID":"73c61954-9500-4f61-aee9-6caa8c9bd5bf","Type":"ContainerStarted","Data":"4c882796b32d8ad310a800ce4c4d8f4e088aa27b0b06228e41c3a30c66f3e35b"} Jul 23 15:06:23.596355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:23.596313 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4cz88" podStartSLOduration=1.5963004920000001 podStartE2EDuration="1.596300492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:06:23.595345821 +0000 UTC m=+371.188796752" watchObservedRunningTime="2026-07-23 15:06:23.596300492 +0000 UTC m=+371.189751411" Jul 23 15:06:31.012204 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.012170 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td"] Jul 23 15:06:31.033355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.033331 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td"] Jul 23 15:06:31.033474 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.033419 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.037529 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.037512 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 23 15:06:31.038494 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.038470 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 23 15:06:31.038595 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.038543 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-v29lk\"" Jul 23 15:06:31.159709 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.159678 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.159871 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.159713 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bt48\" (UniqueName: \"kubernetes.io/projected/3118cf20-49ec-48fe-a64a-7164a5520d80-kube-api-access-4bt48\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.159871 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.159850 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.261094 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.261066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.261234 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.261108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.261234 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.261126 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bt48\" (UniqueName: \"kubernetes.io/projected/3118cf20-49ec-48fe-a64a-7164a5520d80-kube-api-access-4bt48\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.261518 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.261495 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.261554 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.261518 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.272418 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.272360 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bt48\" (UniqueName: \"kubernetes.io/projected/3118cf20-49ec-48fe-a64a-7164a5520d80-kube-api-access-4bt48\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.342986 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.342963 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:31.502861 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.502839 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td"] Jul 23 15:06:31.504547 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:06:31.504520 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3118cf20_49ec_48fe_a64a_7164a5520d80.slice/crio-a6ba8f36e5a62c328bda88d45024e0fd714ee34482e5f778a4852ed5e3396344 WatchSource:0}: Error finding container a6ba8f36e5a62c328bda88d45024e0fd714ee34482e5f778a4852ed5e3396344: Status 404 returned error can't find the container with id a6ba8f36e5a62c328bda88d45024e0fd714ee34482e5f778a4852ed5e3396344 Jul 23 15:06:31.614458 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:31.614430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" event={"ID":"3118cf20-49ec-48fe-a64a-7164a5520d80","Type":"ContainerStarted","Data":"a6ba8f36e5a62c328bda88d45024e0fd714ee34482e5f778a4852ed5e3396344"} Jul 23 15:06:32.621024 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:32.620917 2568 generic.go:356] "Generic (PLEG): container finished" podID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerID="201e75d13b328b9a4f18309f7e5cb8f6c3ee227a9e2907dc1b0377ee4cc1eb6f" exitCode=0 Jul 23 15:06:32.621392 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:32.621029 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" event={"ID":"3118cf20-49ec-48fe-a64a-7164a5520d80","Type":"ContainerDied","Data":"201e75d13b328b9a4f18309f7e5cb8f6c3ee227a9e2907dc1b0377ee4cc1eb6f"} Jul 23 15:06:33.626781 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:33.626746 2568 generic.go:356] "Generic (PLEG): container finished" podID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerID="0b56052193bb34d924379f0a4a89695fccf864365e09c4d87079bdbda530ab43" exitCode=0 Jul 23 15:06:33.627085 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:33.626822 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" event={"ID":"3118cf20-49ec-48fe-a64a-7164a5520d80","Type":"ContainerDied","Data":"0b56052193bb34d924379f0a4a89695fccf864365e09c4d87079bdbda530ab43"} Jul 23 15:06:34.633059 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:34.633027 2568 generic.go:356] "Generic (PLEG): container finished" podID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerID="d272472d3d4573135f0fbd57fede97029aca413a7d5e26564b992471cc2b2e5e" exitCode=0 Jul 23 15:06:34.633546 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:34.633080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" event={"ID":"3118cf20-49ec-48fe-a64a-7164a5520d80","Type":"ContainerDied","Data":"d272472d3d4573135f0fbd57fede97029aca413a7d5e26564b992471cc2b2e5e"} Jul 23 15:06:35.767688 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:35.767665 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:35.903574 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:35.903521 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-bundle\") pod \"3118cf20-49ec-48fe-a64a-7164a5520d80\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " Jul 23 15:06:35.903679 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:35.903608 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-util\") pod \"3118cf20-49ec-48fe-a64a-7164a5520d80\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " Jul 23 15:06:35.903679 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:35.903640 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bt48\" (UniqueName: \"kubernetes.io/projected/3118cf20-49ec-48fe-a64a-7164a5520d80-kube-api-access-4bt48\") pod \"3118cf20-49ec-48fe-a64a-7164a5520d80\" (UID: \"3118cf20-49ec-48fe-a64a-7164a5520d80\") " Jul 23 15:06:35.904283 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:35.904262 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-bundle" (OuterVolumeSpecName: "bundle") pod "3118cf20-49ec-48fe-a64a-7164a5520d80" (UID: "3118cf20-49ec-48fe-a64a-7164a5520d80"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:06:35.905682 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:35.905653 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3118cf20-49ec-48fe-a64a-7164a5520d80-kube-api-access-4bt48" (OuterVolumeSpecName: "kube-api-access-4bt48") pod "3118cf20-49ec-48fe-a64a-7164a5520d80" (UID: "3118cf20-49ec-48fe-a64a-7164a5520d80"). InnerVolumeSpecName "kube-api-access-4bt48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:06:35.912102 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:35.912066 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-util" (OuterVolumeSpecName: "util") pod "3118cf20-49ec-48fe-a64a-7164a5520d80" (UID: "3118cf20-49ec-48fe-a64a-7164a5520d80"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:06:36.004785 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:36.004763 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-bundle\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:36.004785 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:36.004783 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3118cf20-49ec-48fe-a64a-7164a5520d80-util\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:36.004900 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:36.004793 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bt48\" (UniqueName: \"kubernetes.io/projected/3118cf20-49ec-48fe-a64a-7164a5520d80-kube-api-access-4bt48\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:06:36.643035 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:36.642982 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" Jul 23 15:06:36.643035 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:36.643018 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvp9td" event={"ID":"3118cf20-49ec-48fe-a64a-7164a5520d80","Type":"ContainerDied","Data":"a6ba8f36e5a62c328bda88d45024e0fd714ee34482e5f778a4852ed5e3396344"} Jul 23 15:06:36.643237 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:36.643070 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ba8f36e5a62c328bda88d45024e0fd714ee34482e5f778a4852ed5e3396344" Jul 23 15:06:48.295325 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295294 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6"] Jul 23 15:06:48.295745 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295586 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerName="pull" Jul 23 15:06:48.295745 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295597 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerName="pull" Jul 23 15:06:48.295745 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295606 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerName="extract" Jul 23 15:06:48.295745 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295611 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerName="extract" Jul 23 15:06:48.295745 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295620 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerName="util" Jul 23 15:06:48.295745 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295626 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerName="util" Jul 23 15:06:48.295745 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.295686 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3118cf20-49ec-48fe-a64a-7164a5520d80" containerName="extract" Jul 23 15:06:48.345201 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.345166 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6"] Jul 23 15:06:48.345383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.345277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.348811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.348776 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 15:06:48.348811 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.348794 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 15:06:48.349047 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.348849 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 15:06:48.349047 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.348940 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zbq89\"" Jul 23 15:06:48.349277 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.349262 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 15:06:48.396054 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.396027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwd9k\" (UniqueName: \"kubernetes.io/projected/bb4b2618-c991-417e-bbec-8e4095c6ba54-kube-api-access-vwd9k\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.396168 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.396087 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2618-c991-417e-bbec-8e4095c6ba54-tmp\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.396168 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.396121 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb4b2618-c991-417e-bbec-8e4095c6ba54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.396292 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.396175 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4b2618-c991-417e-bbec-8e4095c6ba54-webhook-cert\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.396292 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.396229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2618-c991-417e-bbec-8e4095c6ba54-manifests\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.497176 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.497142 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwd9k\" (UniqueName: \"kubernetes.io/projected/bb4b2618-c991-417e-bbec-8e4095c6ba54-kube-api-access-vwd9k\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.497311 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.497194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2618-c991-417e-bbec-8e4095c6ba54-tmp\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.497311 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.497230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb4b2618-c991-417e-bbec-8e4095c6ba54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.497311 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.497254 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4b2618-c991-417e-bbec-8e4095c6ba54-webhook-cert\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.497311 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.497279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2618-c991-417e-bbec-8e4095c6ba54-manifests\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.497715 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.497688 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2618-c991-417e-bbec-8e4095c6ba54-manifests\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.499727 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.499702 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/bb4b2618-c991-417e-bbec-8e4095c6ba54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.499909 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.499891 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb4b2618-c991-417e-bbec-8e4095c6ba54-tmp\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.500175 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.500157 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4b2618-c991-417e-bbec-8e4095c6ba54-webhook-cert\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.514357 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.514335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwd9k\" (UniqueName: \"kubernetes.io/projected/bb4b2618-c991-417e-bbec-8e4095c6ba54-kube-api-access-vwd9k\") pod \"opendatahub-operator-controller-manager-75984d86c4-l4xn6\" (UID: \"bb4b2618-c991-417e-bbec-8e4095c6ba54\") " pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.656121 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.656068 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:48.818676 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:48.818652 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6"] Jul 23 15:06:48.820411 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:06:48.820385 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4b2618_c991_417e_bbec_8e4095c6ba54.slice/crio-9f9c6770cb4e3b844f1fa398e7f5a714d953cd363e1daa976606e7213fe8d9b3 WatchSource:0}: Error finding container 9f9c6770cb4e3b844f1fa398e7f5a714d953cd363e1daa976606e7213fe8d9b3: Status 404 returned error can't find the container with id 9f9c6770cb4e3b844f1fa398e7f5a714d953cd363e1daa976606e7213fe8d9b3 Jul 23 15:06:49.707314 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:49.707271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" event={"ID":"bb4b2618-c991-417e-bbec-8e4095c6ba54","Type":"ContainerStarted","Data":"9f9c6770cb4e3b844f1fa398e7f5a714d953cd363e1daa976606e7213fe8d9b3"} Jul 23 15:06:52.727262 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:52.727224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" event={"ID":"bb4b2618-c991-417e-bbec-8e4095c6ba54","Type":"ContainerStarted","Data":"12a79c2aac652ba653de237ca4dc4b24c2785a4b0aab02993f0a9f1fdfd81572"} Jul 23 15:06:53.733631 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:53.733598 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb4b2618-c991-417e-bbec-8e4095c6ba54" containerID="12a79c2aac652ba653de237ca4dc4b24c2785a4b0aab02993f0a9f1fdfd81572" exitCode=0 Jul 23 15:06:53.734244 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:53.733678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" event={"ID":"bb4b2618-c991-417e-bbec-8e4095c6ba54","Type":"ContainerDied","Data":"12a79c2aac652ba653de237ca4dc4b24c2785a4b0aab02993f0a9f1fdfd81572"} Jul 23 15:06:54.741598 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:54.741564 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" event={"ID":"bb4b2618-c991-417e-bbec-8e4095c6ba54","Type":"ContainerStarted","Data":"98acb0b1583aa2c3f61b446501e389ffb3f8b7868bb77ea4c119df9d35580694"} Jul 23 15:06:54.742095 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:54.741703 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:06:54.772024 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:06:54.771965 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" podStartSLOduration=2.061652225 podStartE2EDuration="6.77195303s" totalImagesPullingTime="3.705485167s" totalInitContainerRuntime="1.004815638s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:06:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:06:54.769452222 +0000 UTC m=+402.362903139" watchObservedRunningTime="2026-07-23 15:06:54.77195303 +0000 UTC m=+402.365403948" Jul 23 15:07:05.748576 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:05.748545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-75984d86c4-l4xn6" Jul 23 15:07:08.461105 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.461073 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-848d77456-5lchx"] Jul 23 15:07:08.464321 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.464305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.469211 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.469191 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 15:07:08.469538 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.469507 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 15:07:08.469796 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.469780 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 15:07:08.470771 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.470680 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 15:07:08.470771 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.470730 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pd9hw\"" Jul 23 15:07:08.482206 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.482184 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848d77456-5lchx"] Jul 23 15:07:08.550765 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.550730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c383a82b-38ec-42dd-9be0-de48c82d87e0-tmp\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.550963 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.550780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c383a82b-38ec-42dd-9be0-de48c82d87e0-tls-certs\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.550963 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.550881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96wsv\" (UniqueName: \"kubernetes.io/projected/c383a82b-38ec-42dd-9be0-de48c82d87e0-kube-api-access-96wsv\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.651312 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.651281 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96wsv\" (UniqueName: \"kubernetes.io/projected/c383a82b-38ec-42dd-9be0-de48c82d87e0-kube-api-access-96wsv\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.651484 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.651332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c383a82b-38ec-42dd-9be0-de48c82d87e0-tmp\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.651484 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.651371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c383a82b-38ec-42dd-9be0-de48c82d87e0-tls-certs\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.651484 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:07:08.651466 2568 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 23 15:07:08.651586 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:07:08.651538 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c383a82b-38ec-42dd-9be0-de48c82d87e0-tls-certs podName:c383a82b-38ec-42dd-9be0-de48c82d87e0 nodeName:}" failed. No retries permitted until 2026-07-23 15:07:09.151518719 +0000 UTC m=+416.744969618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c383a82b-38ec-42dd-9be0-de48c82d87e0-tls-certs") pod "kube-auth-proxy-848d77456-5lchx" (UID: "c383a82b-38ec-42dd-9be0-de48c82d87e0") : secret "kube-auth-proxy-tls" not found Jul 23 15:07:08.653652 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.653633 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c383a82b-38ec-42dd-9be0-de48c82d87e0-tmp\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.664774 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.664752 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96wsv\" (UniqueName: \"kubernetes.io/projected/c383a82b-38ec-42dd-9be0-de48c82d87e0-kube-api-access-96wsv\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:08.861610 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.861530 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gkw95"] Jul 23 15:07:08.864983 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.864964 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:08.886131 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.886108 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 15:07:08.886264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.886211 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pnq6z\"" Jul 23 15:07:08.893519 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.893488 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gkw95"] Jul 23 15:07:08.953751 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.953717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:08.953913 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:08.953756 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzrh\" (UniqueName: \"kubernetes.io/projected/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-kube-api-access-rzzrh\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:09.054368 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.054330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:09.054555 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:07:09.054431 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 15:07:09.054555 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:07:09.054503 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert podName:637001c1-bdf5-44ca-ad5e-4404f6af2c9d nodeName:}" failed. No retries permitted until 2026-07-23 15:07:09.554483286 +0000 UTC m=+417.147934187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert") pod "odh-model-controller-7998c67dc4-gkw95" (UID: "637001c1-bdf5-44ca-ad5e-4404f6af2c9d") : secret "odh-model-controller-webhook-cert" not found Jul 23 15:07:09.054555 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.054522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzrh\" (UniqueName: \"kubernetes.io/projected/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-kube-api-access-rzzrh\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:09.065654 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.065629 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzrh\" (UniqueName: \"kubernetes.io/projected/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-kube-api-access-rzzrh\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:09.155323 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.155232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c383a82b-38ec-42dd-9be0-de48c82d87e0-tls-certs\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:09.158116 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.158091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c383a82b-38ec-42dd-9be0-de48c82d87e0-tls-certs\") pod \"kube-auth-proxy-848d77456-5lchx\" (UID: \"c383a82b-38ec-42dd-9be0-de48c82d87e0\") " pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:09.374126 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.374082 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" Jul 23 15:07:09.559191 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.559153 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:09.559606 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:07:09.559346 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 23 15:07:09.559606 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:07:09.559410 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert podName:637001c1-bdf5-44ca-ad5e-4404f6af2c9d nodeName:}" failed. No retries permitted until 2026-07-23 15:07:10.559392078 +0000 UTC m=+418.152842985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert") pod "odh-model-controller-7998c67dc4-gkw95" (UID: "637001c1-bdf5-44ca-ad5e-4404f6af2c9d") : secret "odh-model-controller-webhook-cert" not found Jul 23 15:07:09.618095 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.618063 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-848d77456-5lchx"] Jul 23 15:07:09.619564 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:07:09.619534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc383a82b_38ec_42dd_9be0_de48c82d87e0.slice/crio-c8bd35b61d5e348150dd86e2c9cb8bb15abfea037fbef147bb8badab33a1e2b5 WatchSource:0}: Error finding container c8bd35b61d5e348150dd86e2c9cb8bb15abfea037fbef147bb8badab33a1e2b5: Status 404 returned error can't find the container with id c8bd35b61d5e348150dd86e2c9cb8bb15abfea037fbef147bb8badab33a1e2b5 Jul 23 15:07:09.810958 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:09.810923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" event={"ID":"c383a82b-38ec-42dd-9be0-de48c82d87e0","Type":"ContainerStarted","Data":"c8bd35b61d5e348150dd86e2c9cb8bb15abfea037fbef147bb8badab33a1e2b5"} Jul 23 15:07:10.438580 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.438547 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dffjv"] Jul 23 15:07:10.443003 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.442970 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.446164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.446147 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nnwpt\"" Jul 23 15:07:10.447159 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.447146 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 15:07:10.457205 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.457181 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dffjv"] Jul 23 15:07:10.567709 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.567677 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7gx2\" (UniqueName: \"kubernetes.io/projected/33a8aab2-13bd-4f1e-b053-4287cfdd5ad9-kube-api-access-l7gx2\") pod \"model-serving-api-54d89bd5fc-dffjv\" (UID: \"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9\") " pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.568234 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.567726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:10.568234 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.567813 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a8aab2-13bd-4f1e-b053-4287cfdd5ad9-tls-certs\") pod \"model-serving-api-54d89bd5fc-dffjv\" (UID: \"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9\") " pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.570650 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.570625 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/637001c1-bdf5-44ca-ad5e-4404f6af2c9d-cert\") pod \"odh-model-controller-7998c67dc4-gkw95\" (UID: \"637001c1-bdf5-44ca-ad5e-4404f6af2c9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:10.668618 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.668585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a8aab2-13bd-4f1e-b053-4287cfdd5ad9-tls-certs\") pod \"model-serving-api-54d89bd5fc-dffjv\" (UID: \"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9\") " pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.668814 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.668653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7gx2\" (UniqueName: \"kubernetes.io/projected/33a8aab2-13bd-4f1e-b053-4287cfdd5ad9-kube-api-access-l7gx2\") pod \"model-serving-api-54d89bd5fc-dffjv\" (UID: \"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9\") " pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.671375 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.671334 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33a8aab2-13bd-4f1e-b053-4287cfdd5ad9-tls-certs\") pod \"model-serving-api-54d89bd5fc-dffjv\" (UID: \"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9\") " pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.681130 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.681104 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:10.709057 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.709031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7gx2\" (UniqueName: \"kubernetes.io/projected/33a8aab2-13bd-4f1e-b053-4287cfdd5ad9-kube-api-access-l7gx2\") pod \"model-serving-api-54d89bd5fc-dffjv\" (UID: \"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9\") " pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.753225 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.753143 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:10.859930 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.859897 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq"] Jul 23 15:07:10.863617 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.863592 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:10.867022 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.866979 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 15:07:10.867162 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.867148 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 15:07:10.867270 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.867252 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k2f8h\"" Jul 23 15:07:10.872292 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.872062 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 15:07:10.872552 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.872437 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 15:07:10.875822 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.875646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gkw95"] Jul 23 15:07:10.880205 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:07:10.880103 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637001c1_bdf5_44ca_ad5e_4404f6af2c9d.slice/crio-7e6d13e6f578fddb5165bbb9b0a378a4c4603a97254c7c980dd7c9c3c3b00f20 WatchSource:0}: Error finding container 7e6d13e6f578fddb5165bbb9b0a378a4c4603a97254c7c980dd7c9c3c3b00f20: Status 404 returned error can't find the container with id 7e6d13e6f578fddb5165bbb9b0a378a4c4603a97254c7c980dd7c9c3c3b00f20 Jul 23 15:07:10.884452 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.884390 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq"] Jul 23 15:07:10.938806 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.938728 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dffjv"] Jul 23 15:07:10.941168 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:07:10.941138 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a8aab2_13bd_4f1e_b053_4287cfdd5ad9.slice/crio-1041a9468e8395f83fcf75f0f95998ffa9810d1d4302d690a3dee3b5c622648f WatchSource:0}: Error finding container 1041a9468e8395f83fcf75f0f95998ffa9810d1d4302d690a3dee3b5c622648f: Status 404 returned error can't find the container with id 1041a9468e8395f83fcf75f0f95998ffa9810d1d4302d690a3dee3b5c622648f Jul 23 15:07:10.971082 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.971053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-cacerts\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:10.971223 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.971093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-token\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:10.971223 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.971152 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:10.971223 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.971169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89gb\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-kube-api-access-t89gb\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:10.971327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.971223 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-local-certs\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:10.971327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.971264 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:10.971327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:10.971312 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.072050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.071982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.072258 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.072074 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89gb\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-kube-api-access-t89gb\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.072258 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.072110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-local-certs\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.072258 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.072149 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.072258 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.072197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.072258 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.072244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-cacerts\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.072513 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.072279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-token\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.073035 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.072951 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.074887 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.074860 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-cacerts\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.075008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.074921 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-local-certs\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.075008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.074939 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.075008 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.074963 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.082242 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.082214 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89gb\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-kube-api-access-t89gb\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.082315 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.082297 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-token\") pod \"istiod-openshift-gateway-7574459656-l7wwq\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.176644 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.176604 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:11.330826 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.330795 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq"] Jul 23 15:07:11.332550 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:07:11.332522 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388f9e4d_0ab1_45ef_aaaa_d2b2f3bc6229.slice/crio-8e835bd6306a65fd9df88abcdc26c9fe3386e26513030fd34f830819df0083c8 WatchSource:0}: Error finding container 8e835bd6306a65fd9df88abcdc26c9fe3386e26513030fd34f830819df0083c8: Status 404 returned error can't find the container with id 8e835bd6306a65fd9df88abcdc26c9fe3386e26513030fd34f830819df0083c8 Jul 23 15:07:11.828010 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.827960 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" event={"ID":"637001c1-bdf5-44ca-ad5e-4404f6af2c9d","Type":"ContainerStarted","Data":"7e6d13e6f578fddb5165bbb9b0a378a4c4603a97254c7c980dd7c9c3c3b00f20"} Jul 23 15:07:11.829453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.829396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" event={"ID":"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229","Type":"ContainerStarted","Data":"8e835bd6306a65fd9df88abcdc26c9fe3386e26513030fd34f830819df0083c8"} Jul 23 15:07:11.830750 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:11.830713 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" event={"ID":"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9","Type":"ContainerStarted","Data":"1041a9468e8395f83fcf75f0f95998ffa9810d1d4302d690a3dee3b5c622648f"} Jul 23 15:07:12.839295 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:12.839114 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" event={"ID":"c383a82b-38ec-42dd-9be0-de48c82d87e0","Type":"ContainerStarted","Data":"744f27151724be2a2852e732821929aec5c47b9b346a2c18354922559b34c053"} Jul 23 15:07:12.871721 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:12.871658 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-848d77456-5lchx" podStartSLOduration=2.048627534 podStartE2EDuration="4.871638972s" totalImagesPullingTime="2.823011438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:07:12.870545055 +0000 UTC m=+420.463995974" watchObservedRunningTime="2026-07-23 15:07:12.871638972 +0000 UTC m=+420.465089892" Jul 23 15:07:16.119148 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.119098 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 15:07:16.119539 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.119174 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 23 15:07:16.862835 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.862791 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" event={"ID":"637001c1-bdf5-44ca-ad5e-4404f6af2c9d","Type":"ContainerStarted","Data":"336e65e89cc301b38e2b7fb368717f8813c2459d2c25011f839d05ba75794809"} Jul 23 15:07:16.863121 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.862911 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:16.864391 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.864370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" event={"ID":"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229","Type":"ContainerStarted","Data":"1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18"} Jul 23 15:07:16.864501 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.864420 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:16.865908 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.865885 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" event={"ID":"33a8aab2-13bd-4f1e-b053-4287cfdd5ad9","Type":"ContainerStarted","Data":"0cdde0717b4ba9d8d53cf8b69d730f381d5bce4ae650af6cab3a58880eeaf3ea"} Jul 23 15:07:16.866055 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.865935 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:16.916673 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.916619 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" podStartSLOduration=3.665926024 podStartE2EDuration="8.916602839s" totalImagesPullingTime="5.250676815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:07:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:07:16.916305962 +0000 UTC m=+424.509756903" watchObservedRunningTime="2026-07-23 15:07:16.916602839 +0000 UTC m=+424.510053758" Jul 23 15:07:16.950015 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.946819 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" podStartSLOduration=1.7671658350000001 podStartE2EDuration="6.946799545s" totalImagesPullingTime="5.17963371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:07:16.943215287 +0000 UTC m=+424.536666207" watchObservedRunningTime="2026-07-23 15:07:16.946799545 +0000 UTC m=+424.540250465" Jul 23 15:07:16.982579 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:16.982518 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" podStartSLOduration=2.198884645 podStartE2EDuration="6.982498953s" totalImagesPullingTime="4.783614308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:07:16.982486225 +0000 UTC m=+424.575937143" watchObservedRunningTime="2026-07-23 15:07:16.982498953 +0000 UTC m=+424.575949874" Jul 23 15:07:17.872898 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:17.872868 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-l7wwq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 15:07:17.873404 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:17.872925 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" podUID="388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 15:07:20.384513 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.384479 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp"] Jul 23 15:07:20.388189 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.388170 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.392630 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.392607 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 23 15:07:20.392936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.392904 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 23 15:07:20.392936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.392913 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 15:07:20.392936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.392921 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 23 15:07:20.393196 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.392980 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 15:07:20.393196 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.392980 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tpsqj\"" Jul 23 15:07:20.404097 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.404072 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp"] Jul 23 15:07:20.468832 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.468798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4882f36-f4e0-4f47-a9f4-00b9822fded6-metrics-cert\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.469070 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.468868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d4882f36-f4e0-4f47-a9f4-00b9822fded6-manager-config\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.469070 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.469023 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfnmh\" (UniqueName: \"kubernetes.io/projected/d4882f36-f4e0-4f47-a9f4-00b9822fded6-kube-api-access-vfnmh\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.469070 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.469066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4882f36-f4e0-4f47-a9f4-00b9822fded6-cert\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.569634 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.569592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4882f36-f4e0-4f47-a9f4-00b9822fded6-metrics-cert\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.569849 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.569660 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d4882f36-f4e0-4f47-a9f4-00b9822fded6-manager-config\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.569849 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.569714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnmh\" (UniqueName: \"kubernetes.io/projected/d4882f36-f4e0-4f47-a9f4-00b9822fded6-kube-api-access-vfnmh\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.569849 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.569734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4882f36-f4e0-4f47-a9f4-00b9822fded6-cert\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.570369 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.570346 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d4882f36-f4e0-4f47-a9f4-00b9822fded6-manager-config\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.572455 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.572429 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d4882f36-f4e0-4f47-a9f4-00b9822fded6-metrics-cert\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.572543 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.572454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4882f36-f4e0-4f47-a9f4-00b9822fded6-cert\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.585267 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.585246 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnmh\" (UniqueName: \"kubernetes.io/projected/d4882f36-f4e0-4f47-a9f4-00b9822fded6-kube-api-access-vfnmh\") pod \"lws-controller-manager-68b858458d-nm2vp\" (UID: \"d4882f36-f4e0-4f47-a9f4-00b9822fded6\") " pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.697796 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.697767 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:20.866641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.866609 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp"] Jul 23 15:07:20.867345 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:07:20.867316 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4882f36_f4e0_4f47_a9f4_00b9822fded6.slice/crio-464e752ddd7c59da586151d8c14c6231ef25f7ca1d7317aa801e1fef70d493be WatchSource:0}: Error finding container 464e752ddd7c59da586151d8c14c6231ef25f7ca1d7317aa801e1fef70d493be: Status 404 returned error can't find the container with id 464e752ddd7c59da586151d8c14c6231ef25f7ca1d7317aa801e1fef70d493be Jul 23 15:07:20.872736 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.872709 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:20.891902 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:20.891875 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" event={"ID":"d4882f36-f4e0-4f47-a9f4-00b9822fded6","Type":"ContainerStarted","Data":"464e752ddd7c59da586151d8c14c6231ef25f7ca1d7317aa801e1fef70d493be"} Jul 23 15:07:22.903142 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:22.903102 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" event={"ID":"d4882f36-f4e0-4f47-a9f4-00b9822fded6","Type":"ContainerStarted","Data":"0d70703db9dde2a763d37d5ca3dd771040aade7052c37e4986b9e2e7458cef68"} Jul 23 15:07:22.903546 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:22.903203 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:07:22.914593 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:22.914564 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq"] Jul 23 15:07:22.914746 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:22.914727 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" podUID="388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" containerName="discovery" containerID="cri-o://1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18" gracePeriod=30 Jul 23 15:07:22.934742 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:22.934674 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" podStartSLOduration=1.49451494 podStartE2EDuration="2.934658313s" totalImagesPullingTime="1.440143373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:07:22.934310683 +0000 UTC m=+430.527761603" watchObservedRunningTime="2026-07-23 15:07:22.934658313 +0000 UTC m=+430.528109232" Jul 23 15:07:23.156164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.156106 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:23.294400 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.294364 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-cacerts\") pod \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " Jul 23 15:07:23.294587 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.294430 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t89gb\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-kube-api-access-t89gb\") pod \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " Jul 23 15:07:23.294587 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.294462 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-local-certs\") pod \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " Jul 23 15:07:23.294587 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.294499 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-token\") pod \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " Jul 23 15:07:23.294587 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.294516 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-kubeconfig\") pod \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " Jul 23 15:07:23.294587 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.294549 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-ca-configmap\") pod \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " Jul 23 15:07:23.294857 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.294589 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-dns-cert\") pod \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\" (UID: \"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229\") " Jul 23 15:07:23.295054 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.295021 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" (UID: "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 15:07:23.296788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.296755 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-local-certs" (OuterVolumeSpecName: "local-certs") pod "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" (UID: "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:07:23.296897 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.296824 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-token" (OuterVolumeSpecName: "istio-token") pod "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" (UID: "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:07:23.297026 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.296984 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-kube-api-access-t89gb" (OuterVolumeSpecName: "kube-api-access-t89gb") pod "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" (UID: "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229"). InnerVolumeSpecName "kube-api-access-t89gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:07:23.297026 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.296986 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-cacerts" (OuterVolumeSpecName: "cacerts") pod "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" (UID: "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:07:23.297173 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.297155 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" (UID: "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:07:23.297229 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.297176 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" (UID: "388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:07:23.396050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.396008 2568 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-local-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:07:23.396050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.396037 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-token\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:07:23.396050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.396046 2568 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-kubeconfig\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:07:23.396050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.396056 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-ca-configmap\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:07:23.396320 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.396066 2568 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-istio-csr-dns-cert\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:07:23.396320 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.396076 2568 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-cacerts\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:07:23.396320 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.396084 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t89gb\" (UniqueName: \"kubernetes.io/projected/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229-kube-api-access-t89gb\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:07:23.909600 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.909567 2568 generic.go:356] "Generic (PLEG): container finished" podID="388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" containerID="1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18" exitCode=0 Jul 23 15:07:23.910112 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.909605 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" Jul 23 15:07:23.910112 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.909637 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" event={"ID":"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229","Type":"ContainerDied","Data":"1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18"} Jul 23 15:07:23.910112 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.909687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq" event={"ID":"388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229","Type":"ContainerDied","Data":"8e835bd6306a65fd9df88abcdc26c9fe3386e26513030fd34f830819df0083c8"} Jul 23 15:07:23.910112 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.909710 2568 scope.go:117] "RemoveContainer" containerID="1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18" Jul 23 15:07:23.920527 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.920505 2568 scope.go:117] "RemoveContainer" containerID="1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18" Jul 23 15:07:23.920821 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:07:23.920801 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18\": container with ID starting with 1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18 not found: ID does not exist" containerID="1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18" Jul 23 15:07:23.920891 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.920831 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18"} err="failed to get container status \"1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18\": rpc error: code = NotFound desc = could not find container \"1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18\": container with ID starting with 1f750f9c53c57a4b7a525302029f3f8449207bd8d0cd9f6dbd820d051b994d18 not found: ID does not exist" Jul 23 15:07:23.936938 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.936914 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq"] Jul 23 15:07:23.942322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:23.942302 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-l7wwq"] Jul 23 15:07:24.970046 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:24.970011 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" path="/var/lib/kubelet/pods/388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229/volumes" Jul 23 15:07:27.873982 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:27.873955 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gkw95" Jul 23 15:07:27.877764 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:27.877741 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dffjv" Jul 23 15:07:33.912166 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:07:33.912133 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68b858458d-nm2vp" Jul 23 15:08:25.252902 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.252863 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n"] Jul 23 15:08:25.253396 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.253285 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" containerName="discovery" Jul 23 15:08:25.253396 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.253298 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" containerName="discovery" Jul 23 15:08:25.253396 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.253365 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="388f9e4d-0ab1-45ef-aaaa-d2b2f3bc6229" containerName="discovery" Jul 23 15:08:25.255849 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.255831 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.258931 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.258910 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 15:08:25.259088 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.258911 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 15:08:25.259838 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.259819 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k7lbl\"" Jul 23 15:08:25.267094 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.267070 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n"] Jul 23 15:08:25.384964 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.384922 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-428df\" (UniqueName: \"kubernetes.io/projected/05cf2b9c-944b-4e6a-8a08-315e1a603359-kube-api-access-428df\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.385161 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.384976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.385161 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.385081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.485768 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.485731 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-428df\" (UniqueName: \"kubernetes.io/projected/05cf2b9c-944b-4e6a-8a08-315e1a603359-kube-api-access-428df\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.485943 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.485789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.485943 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.485818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.486248 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.486230 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.486287 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.486261 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.495847 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.495820 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-428df\" (UniqueName: \"kubernetes.io/projected/05cf2b9c-944b-4e6a-8a08-315e1a603359-kube-api-access-428df\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.564695 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.564611 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:25.703466 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.703439 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n"] Jul 23 15:08:25.705200 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:08:25.705174 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cf2b9c_944b_4e6a_8a08_315e1a603359.slice/crio-583e5ea0588c8b3cafe48170da55e12798500260e0e69fc8f25829ea83b0e8c4 WatchSource:0}: Error finding container 583e5ea0588c8b3cafe48170da55e12798500260e0e69fc8f25829ea83b0e8c4: Status 404 returned error can't find the container with id 583e5ea0588c8b3cafe48170da55e12798500260e0e69fc8f25829ea83b0e8c4 Jul 23 15:08:25.812687 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.812659 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx"] Jul 23 15:08:25.815291 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.815251 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.830242 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.830215 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx"] Jul 23 15:08:25.889188 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.889157 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2nd\" (UniqueName: \"kubernetes.io/projected/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-kube-api-access-xx2nd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.889333 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.889214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.889333 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.889276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.990343 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.990309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2nd\" (UniqueName: \"kubernetes.io/projected/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-kube-api-access-xx2nd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.990511 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.990367 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.990511 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.990396 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.990743 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.990725 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:25.990819 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:25.990801 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:26.000693 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.000671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2nd\" (UniqueName: \"kubernetes.io/projected/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-kube-api-access-xx2nd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:26.161752 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.161671 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:26.211637 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.211586 2568 generic.go:356] "Generic (PLEG): container finished" podID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerID="83b9453f158e3a46804fffddb41db1a0151af7657ef7f5d9c6bfe199dea625df" exitCode=0 Jul 23 15:08:26.211806 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.211641 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" event={"ID":"05cf2b9c-944b-4e6a-8a08-315e1a603359","Type":"ContainerDied","Data":"83b9453f158e3a46804fffddb41db1a0151af7657ef7f5d9c6bfe199dea625df"} Jul 23 15:08:26.211806 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.211714 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" event={"ID":"05cf2b9c-944b-4e6a-8a08-315e1a603359","Type":"ContainerStarted","Data":"583e5ea0588c8b3cafe48170da55e12798500260e0e69fc8f25829ea83b0e8c4"} Jul 23 15:08:26.301910 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.301885 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx"] Jul 23 15:08:26.434404 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.432697 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95"] Jul 23 15:08:26.436686 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.436665 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.443747 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.443720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95"] Jul 23 15:08:26.495463 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.495432 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.495610 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.495480 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5467\" (UniqueName: \"kubernetes.io/projected/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-kube-api-access-r5467\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.495610 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.495592 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.596859 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.596817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.597069 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.596907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.597069 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.596952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5467\" (UniqueName: \"kubernetes.io/projected/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-kube-api-access-r5467\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.597282 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.597263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.597316 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.597294 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.607244 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.607216 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5467\" (UniqueName: \"kubernetes.io/projected/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-kube-api-access-r5467\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.775459 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.775428 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:26.838556 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.838526 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5"] Jul 23 15:08:26.841227 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.841204 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:26.855147 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.855120 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5"] Jul 23 15:08:26.900199 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.900165 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:26.900379 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.900281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:26.900379 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.900370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfznr\" (UniqueName: \"kubernetes.io/projected/a2549618-c352-44a2-8e00-5b50a5f38df3-kube-api-access-lfznr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:26.919383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:26.919357 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95"] Jul 23 15:08:26.920786 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:08:26.920756 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cb3764_8f3c_4b6c_8ed8_2734c0c08061.slice/crio-09c850d954764b91879add82740702db5481e1a568122d9341ae13e129dd5e9e WatchSource:0}: Error finding container 09c850d954764b91879add82740702db5481e1a568122d9341ae13e129dd5e9e: Status 404 returned error can't find the container with id 09c850d954764b91879add82740702db5481e1a568122d9341ae13e129dd5e9e Jul 23 15:08:27.001571 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.001543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfznr\" (UniqueName: \"kubernetes.io/projected/a2549618-c352-44a2-8e00-5b50a5f38df3-kube-api-access-lfznr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:27.001669 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.001589 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:27.001669 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.001650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:27.002060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.002040 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:27.002145 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.002065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:27.013190 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.013166 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfznr\" (UniqueName: \"kubernetes.io/projected/a2549618-c352-44a2-8e00-5b50a5f38df3-kube-api-access-lfznr\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:27.152758 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.152728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:27.218188 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.218160 2568 generic.go:356] "Generic (PLEG): container finished" podID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerID="321bfe1e7ab92f542b91bf3052f68338e72c1042ae4a7c7220595445a8f65358" exitCode=0 Jul 23 15:08:27.218338 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.218247 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" event={"ID":"05cf2b9c-944b-4e6a-8a08-315e1a603359","Type":"ContainerDied","Data":"321bfe1e7ab92f542b91bf3052f68338e72c1042ae4a7c7220595445a8f65358"} Jul 23 15:08:27.219934 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.219804 2568 generic.go:356] "Generic (PLEG): container finished" podID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerID="0f64851668711bed354c9d3819a89044b7860f8168bd69eac2d44d8e4ec639ef" exitCode=0 Jul 23 15:08:27.219934 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.219885 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" event={"ID":"37cb3764-8f3c-4b6c-8ed8-2734c0c08061","Type":"ContainerDied","Data":"0f64851668711bed354c9d3819a89044b7860f8168bd69eac2d44d8e4ec639ef"} Jul 23 15:08:27.220152 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.219939 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" event={"ID":"37cb3764-8f3c-4b6c-8ed8-2734c0c08061","Type":"ContainerStarted","Data":"09c850d954764b91879add82740702db5481e1a568122d9341ae13e129dd5e9e"} Jul 23 15:08:27.221574 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.221554 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerID="e3b3dc215e524c9a49dd3caecd3cedfc268ff56ca6e3fd33d593e21ae7803620" exitCode=0 Jul 23 15:08:27.221689 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.221640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" event={"ID":"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c","Type":"ContainerDied","Data":"e3b3dc215e524c9a49dd3caecd3cedfc268ff56ca6e3fd33d593e21ae7803620"} Jul 23 15:08:27.221754 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.221689 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" event={"ID":"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c","Type":"ContainerStarted","Data":"671bb8a2e15c77da30b9cdf2d153354ce21ec58cb542fb2c6231c394f2891d6e"} Jul 23 15:08:27.303407 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:27.303323 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5"] Jul 23 15:08:27.307748 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:08:27.307720 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2549618_c352_44a2_8e00_5b50a5f38df3.slice/crio-7016bed56c5dcda9a6c3b920a852cb32a722ce3414f7175b6c8a9807ba04a6a7 WatchSource:0}: Error finding container 7016bed56c5dcda9a6c3b920a852cb32a722ce3414f7175b6c8a9807ba04a6a7: Status 404 returned error can't find the container with id 7016bed56c5dcda9a6c3b920a852cb32a722ce3414f7175b6c8a9807ba04a6a7 Jul 23 15:08:28.229598 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:28.229505 2568 generic.go:356] "Generic (PLEG): container finished" podID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerID="3252c4f3df94bba91e598da453f0444e1d572c2fee0afd79057a83a11dba3dee" exitCode=0 Jul 23 15:08:28.229770 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:28.229594 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" event={"ID":"05cf2b9c-944b-4e6a-8a08-315e1a603359","Type":"ContainerDied","Data":"3252c4f3df94bba91e598da453f0444e1d572c2fee0afd79057a83a11dba3dee"} Jul 23 15:08:28.231366 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:28.231341 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerID="aa14eda386e716b6e5063dbc55be1ad63bb87b9939f5cc47fbcf345ee0cd1f90" exitCode=0 Jul 23 15:08:28.231510 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:28.231428 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" event={"ID":"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c","Type":"ContainerDied","Data":"aa14eda386e716b6e5063dbc55be1ad63bb87b9939f5cc47fbcf345ee0cd1f90"} Jul 23 15:08:28.233053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:28.232917 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerID="87d1fbd12b3938dd22dca525ef4f8025c7520939c5fd839b108bcc1b7ae4a9fd" exitCode=0 Jul 23 15:08:28.233053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:28.233014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" event={"ID":"a2549618-c352-44a2-8e00-5b50a5f38df3","Type":"ContainerDied","Data":"87d1fbd12b3938dd22dca525ef4f8025c7520939c5fd839b108bcc1b7ae4a9fd"} Jul 23 15:08:28.233189 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:28.233056 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" event={"ID":"a2549618-c352-44a2-8e00-5b50a5f38df3","Type":"ContainerStarted","Data":"7016bed56c5dcda9a6c3b920a852cb32a722ce3414f7175b6c8a9807ba04a6a7"} Jul 23 15:08:29.242274 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.242239 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerID="a55eed10ba56a2e6a8749b803c978c4b81852b7afb21fe81925cbfda7d9e603e" exitCode=0 Jul 23 15:08:29.242727 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.242326 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" event={"ID":"a2549618-c352-44a2-8e00-5b50a5f38df3","Type":"ContainerDied","Data":"a55eed10ba56a2e6a8749b803c978c4b81852b7afb21fe81925cbfda7d9e603e"} Jul 23 15:08:29.244184 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.244162 2568 generic.go:356] "Generic (PLEG): container finished" podID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerID="a9c99d14db710b3b777813746d539151926a01ca53657ce1e245295667bef7e7" exitCode=0 Jul 23 15:08:29.244282 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.244229 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" event={"ID":"37cb3764-8f3c-4b6c-8ed8-2734c0c08061","Type":"ContainerDied","Data":"a9c99d14db710b3b777813746d539151926a01ca53657ce1e245295667bef7e7"} Jul 23 15:08:29.246478 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.246456 2568 generic.go:356] "Generic (PLEG): container finished" podID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerID="5e87787f569cbf77dd8f557e01f2522f8a23430f0a77ca53e6af71cee846a40f" exitCode=0 Jul 23 15:08:29.246547 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.246499 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" event={"ID":"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c","Type":"ContainerDied","Data":"5e87787f569cbf77dd8f557e01f2522f8a23430f0a77ca53e6af71cee846a40f"} Jul 23 15:08:29.373954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.373924 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:29.421734 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.421701 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-bundle\") pod \"05cf2b9c-944b-4e6a-8a08-315e1a603359\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " Jul 23 15:08:29.421890 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.421759 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-util\") pod \"05cf2b9c-944b-4e6a-8a08-315e1a603359\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " Jul 23 15:08:29.421890 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.421834 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-428df\" (UniqueName: \"kubernetes.io/projected/05cf2b9c-944b-4e6a-8a08-315e1a603359-kube-api-access-428df\") pod \"05cf2b9c-944b-4e6a-8a08-315e1a603359\" (UID: \"05cf2b9c-944b-4e6a-8a08-315e1a603359\") " Jul 23 15:08:29.422247 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.422211 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-bundle" (OuterVolumeSpecName: "bundle") pod "05cf2b9c-944b-4e6a-8a08-315e1a603359" (UID: "05cf2b9c-944b-4e6a-8a08-315e1a603359"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:29.424570 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.424542 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cf2b9c-944b-4e6a-8a08-315e1a603359-kube-api-access-428df" (OuterVolumeSpecName: "kube-api-access-428df") pod "05cf2b9c-944b-4e6a-8a08-315e1a603359" (UID: "05cf2b9c-944b-4e6a-8a08-315e1a603359"). InnerVolumeSpecName "kube-api-access-428df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:08:29.427778 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.427752 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-util" (OuterVolumeSpecName: "util") pod "05cf2b9c-944b-4e6a-8a08-315e1a603359" (UID: "05cf2b9c-944b-4e6a-8a08-315e1a603359"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:29.522529 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.522498 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-bundle\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:29.522529 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.522523 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05cf2b9c-944b-4e6a-8a08-315e1a603359-util\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:29.522529 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:29.522532 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-428df\" (UniqueName: \"kubernetes.io/projected/05cf2b9c-944b-4e6a-8a08-315e1a603359-kube-api-access-428df\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:30.254444 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.254414 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" Jul 23 15:08:30.254862 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.254413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n" event={"ID":"05cf2b9c-944b-4e6a-8a08-315e1a603359","Type":"ContainerDied","Data":"583e5ea0588c8b3cafe48170da55e12798500260e0e69fc8f25829ea83b0e8c4"} Jul 23 15:08:30.254862 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.254547 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="583e5ea0588c8b3cafe48170da55e12798500260e0e69fc8f25829ea83b0e8c4" Jul 23 15:08:30.256538 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.256516 2568 generic.go:356] "Generic (PLEG): container finished" podID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerID="f36f9057f88786995b5fbae0d4d898500162305f8dcd29c235b1be676f29d78c" exitCode=0 Jul 23 15:08:30.256653 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.256602 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" event={"ID":"37cb3764-8f3c-4b6c-8ed8-2734c0c08061","Type":"ContainerDied","Data":"f36f9057f88786995b5fbae0d4d898500162305f8dcd29c235b1be676f29d78c"} Jul 23 15:08:30.258687 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.258663 2568 generic.go:356] "Generic (PLEG): container finished" podID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerID="e0a462ad7a181eaca66d5e6253d6af0a72d01b05dfc8ecdd57b4d26115402d81" exitCode=0 Jul 23 15:08:30.258765 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.258741 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" event={"ID":"a2549618-c352-44a2-8e00-5b50a5f38df3","Type":"ContainerDied","Data":"e0a462ad7a181eaca66d5e6253d6af0a72d01b05dfc8ecdd57b4d26115402d81"} Jul 23 15:08:30.398761 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.398738 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:30.530553 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.530468 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx2nd\" (UniqueName: \"kubernetes.io/projected/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-kube-api-access-xx2nd\") pod \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " Jul 23 15:08:30.530553 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.530514 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-util\") pod \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " Jul 23 15:08:30.530744 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.530603 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-bundle\") pod \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\" (UID: \"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c\") " Jul 23 15:08:30.531191 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.531151 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-bundle" (OuterVolumeSpecName: "bundle") pod "7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" (UID: "7a09ffcc-27ad-4e5e-945c-33c19a4cb43c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:30.532680 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.532649 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-kube-api-access-xx2nd" (OuterVolumeSpecName: "kube-api-access-xx2nd") pod "7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" (UID: "7a09ffcc-27ad-4e5e-945c-33c19a4cb43c"). InnerVolumeSpecName "kube-api-access-xx2nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:08:30.535944 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.535913 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-util" (OuterVolumeSpecName: "util") pod "7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" (UID: "7a09ffcc-27ad-4e5e-945c-33c19a4cb43c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:30.631776 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.631742 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-bundle\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:30.631776 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.631773 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx2nd\" (UniqueName: \"kubernetes.io/projected/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-kube-api-access-xx2nd\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:30.631776 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:30.631784 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a09ffcc-27ad-4e5e-945c-33c19a4cb43c-util\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:31.265750 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.265664 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" event={"ID":"7a09ffcc-27ad-4e5e-945c-33c19a4cb43c","Type":"ContainerDied","Data":"671bb8a2e15c77da30b9cdf2d153354ce21ec58cb542fb2c6231c394f2891d6e"} Jul 23 15:08:31.265750 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.265720 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671bb8a2e15c77da30b9cdf2d153354ce21ec58cb542fb2c6231c394f2891d6e" Jul 23 15:08:31.265750 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.265671 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx" Jul 23 15:08:31.426735 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.426713 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:31.429923 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.429899 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:31.539602 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.539528 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-util\") pod \"a2549618-c352-44a2-8e00-5b50a5f38df3\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " Jul 23 15:08:31.539602 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.539568 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-bundle\") pod \"a2549618-c352-44a2-8e00-5b50a5f38df3\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " Jul 23 15:08:31.539602 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.539590 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-util\") pod \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " Jul 23 15:08:31.539877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.539613 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5467\" (UniqueName: \"kubernetes.io/projected/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-kube-api-access-r5467\") pod \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " Jul 23 15:08:31.539877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.539642 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-bundle\") pod \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\" (UID: \"37cb3764-8f3c-4b6c-8ed8-2734c0c08061\") " Jul 23 15:08:31.539877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.539743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfznr\" (UniqueName: \"kubernetes.io/projected/a2549618-c352-44a2-8e00-5b50a5f38df3-kube-api-access-lfznr\") pod \"a2549618-c352-44a2-8e00-5b50a5f38df3\" (UID: \"a2549618-c352-44a2-8e00-5b50a5f38df3\") " Jul 23 15:08:31.540313 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.540237 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-bundle" (OuterVolumeSpecName: "bundle") pod "37cb3764-8f3c-4b6c-8ed8-2734c0c08061" (UID: "37cb3764-8f3c-4b6c-8ed8-2734c0c08061"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:31.540313 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.540288 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-bundle" (OuterVolumeSpecName: "bundle") pod "a2549618-c352-44a2-8e00-5b50a5f38df3" (UID: "a2549618-c352-44a2-8e00-5b50a5f38df3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:31.542224 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.542200 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-kube-api-access-r5467" (OuterVolumeSpecName: "kube-api-access-r5467") pod "37cb3764-8f3c-4b6c-8ed8-2734c0c08061" (UID: "37cb3764-8f3c-4b6c-8ed8-2734c0c08061"). InnerVolumeSpecName "kube-api-access-r5467". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:08:31.542322 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.542294 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2549618-c352-44a2-8e00-5b50a5f38df3-kube-api-access-lfznr" (OuterVolumeSpecName: "kube-api-access-lfznr") pod "a2549618-c352-44a2-8e00-5b50a5f38df3" (UID: "a2549618-c352-44a2-8e00-5b50a5f38df3"). InnerVolumeSpecName "kube-api-access-lfznr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:08:31.546178 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.546155 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-util" (OuterVolumeSpecName: "util") pod "37cb3764-8f3c-4b6c-8ed8-2734c0c08061" (UID: "37cb3764-8f3c-4b6c-8ed8-2734c0c08061"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:31.546415 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.546395 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-util" (OuterVolumeSpecName: "util") pod "a2549618-c352-44a2-8e00-5b50a5f38df3" (UID: "a2549618-c352-44a2-8e00-5b50a5f38df3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:08:31.641253 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.641208 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-util\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:31.641253 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.641245 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2549618-c352-44a2-8e00-5b50a5f38df3-bundle\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:31.641253 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.641257 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-util\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:31.641488 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.641269 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5467\" (UniqueName: \"kubernetes.io/projected/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-kube-api-access-r5467\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:31.641488 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.641282 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/37cb3764-8f3c-4b6c-8ed8-2734c0c08061-bundle\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:31.641488 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:31.641294 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfznr\" (UniqueName: \"kubernetes.io/projected/a2549618-c352-44a2-8e00-5b50a5f38df3-kube-api-access-lfznr\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:08:32.272727 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:32.272699 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" Jul 23 15:08:32.272727 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:32.272708 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95" event={"ID":"37cb3764-8f3c-4b6c-8ed8-2734c0c08061","Type":"ContainerDied","Data":"09c850d954764b91879add82740702db5481e1a568122d9341ae13e129dd5e9e"} Jul 23 15:08:32.273243 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:32.272749 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09c850d954764b91879add82740702db5481e1a568122d9341ae13e129dd5e9e" Jul 23 15:08:32.281043 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:32.281007 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" event={"ID":"a2549618-c352-44a2-8e00-5b50a5f38df3","Type":"ContainerDied","Data":"7016bed56c5dcda9a6c3b920a852cb32a722ce3414f7175b6c8a9807ba04a6a7"} Jul 23 15:08:32.281190 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:32.281060 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5" Jul 23 15:08:32.281190 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:08:32.281079 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7016bed56c5dcda9a6c3b920a852cb32a722ce3414f7175b6c8a9807ba04a6a7" Jul 23 15:09:03.463637 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.463598 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp"] Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464109 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerName="util" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464129 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerName="util" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464142 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerName="pull" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464150 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerName="pull" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464160 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerName="extract" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464169 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerName="extract" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464182 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerName="util" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464189 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerName="util" Jul 23 15:09:03.464195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464199 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464208 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464235 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464244 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464254 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerName="pull" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464261 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerName="pull" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464273 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerName="util" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464282 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerName="util" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464297 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerName="pull" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464305 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerName="pull" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464314 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerName="util" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464321 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerName="util" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464330 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerName="pull" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464339 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerName="pull" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464356 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464363 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464442 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="05cf2b9c-944b-4e6a-8a08-315e1a603359" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464457 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2549618-c352-44a2-8e00-5b50a5f38df3" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464469 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="37cb3764-8f3c-4b6c-8ed8-2734c0c08061" containerName="extract" Jul 23 15:09:03.464625 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.464481 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a09ffcc-27ad-4e5e-945c-33c19a4cb43c" containerName="extract" Jul 23 15:09:03.469301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.469279 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.471939 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.471919 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 23 15:09:03.473497 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.473481 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 15:09:03.473606 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.473578 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 23 15:09:03.474523 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.474504 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k7lbl\"" Jul 23 15:09:03.474578 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.474563 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 15:09:03.479272 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.479250 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp"] Jul 23 15:09:03.586267 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.586243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd2hh\" (UniqueName: \"kubernetes.io/projected/801a2923-1a42-4ba8-8876-4bdf512cfba5-kube-api-access-wd2hh\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.586389 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.586298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/801a2923-1a42-4ba8-8876-4bdf512cfba5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.586389 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.586345 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/801a2923-1a42-4ba8-8876-4bdf512cfba5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.687543 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.687518 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/801a2923-1a42-4ba8-8876-4bdf512cfba5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.687659 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.687558 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd2hh\" (UniqueName: \"kubernetes.io/projected/801a2923-1a42-4ba8-8876-4bdf512cfba5-kube-api-access-wd2hh\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.687659 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.687608 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/801a2923-1a42-4ba8-8876-4bdf512cfba5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.687777 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:09:03.687703 2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 23 15:09:03.687777 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:09:03.687766 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/801a2923-1a42-4ba8-8876-4bdf512cfba5-plugin-serving-cert podName:801a2923-1a42-4ba8-8876-4bdf512cfba5 nodeName:}" failed. No retries permitted until 2026-07-23 15:09:04.187746914 +0000 UTC m=+531.781197811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/801a2923-1a42-4ba8-8876-4bdf512cfba5-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-dncdp" (UID: "801a2923-1a42-4ba8-8876-4bdf512cfba5") : secret "plugin-serving-cert" not found Jul 23 15:09:03.688240 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.688217 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/801a2923-1a42-4ba8-8876-4bdf512cfba5-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:03.698263 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:03.698244 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd2hh\" (UniqueName: \"kubernetes.io/projected/801a2923-1a42-4ba8-8876-4bdf512cfba5-kube-api-access-wd2hh\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:04.191393 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:04.191365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/801a2923-1a42-4ba8-8876-4bdf512cfba5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:04.193771 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:04.193749 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/801a2923-1a42-4ba8-8876-4bdf512cfba5-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dncdp\" (UID: \"801a2923-1a42-4ba8-8876-4bdf512cfba5\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:04.378669 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:04.378643 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" Jul 23 15:09:04.529677 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:04.529606 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp"] Jul 23 15:09:04.532009 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:09:04.531972 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod801a2923_1a42_4ba8_8876_4bdf512cfba5.slice/crio-2d202fd43085a49995f486e304f49ee2ee46e4421b08229ee1be3b7fc11ce701 WatchSource:0}: Error finding container 2d202fd43085a49995f486e304f49ee2ee46e4421b08229ee1be3b7fc11ce701: Status 404 returned error can't find the container with id 2d202fd43085a49995f486e304f49ee2ee46e4421b08229ee1be3b7fc11ce701 Jul 23 15:09:05.445790 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:05.445739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" event={"ID":"801a2923-1a42-4ba8-8876-4bdf512cfba5","Type":"ContainerStarted","Data":"2d202fd43085a49995f486e304f49ee2ee46e4421b08229ee1be3b7fc11ce701"} Jul 23 15:09:26.470383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.470350 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm"] Jul 23 15:09:26.545576 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.545548 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm"] Jul 23 15:09:26.545735 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.545641 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:26.551746 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.551724 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cddmh\"" Jul 23 15:09:26.700075 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.700039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g8xpm\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:26.700228 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.700138 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcqj\" (UniqueName: \"kubernetes.io/projected/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-kube-api-access-fzcqj\") pod \"kuadrant-operator-controller-manager-86878c769f-g8xpm\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:26.801104 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.801033 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcqj\" (UniqueName: \"kubernetes.io/projected/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-kube-api-access-fzcqj\") pod \"kuadrant-operator-controller-manager-86878c769f-g8xpm\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:26.801271 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.801100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g8xpm\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:26.801437 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.801416 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g8xpm\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:26.827182 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.827149 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcqj\" (UniqueName: \"kubernetes.io/projected/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-kube-api-access-fzcqj\") pod \"kuadrant-operator-controller-manager-86878c769f-g8xpm\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:26.855439 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:26.855415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:27.422190 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:27.422169 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm"] Jul 23 15:09:27.422943 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:09:27.422921 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c88d74d_5ff6_47da_ab89_0ba978b84ee7.slice/crio-749f03782329967aa6e361b31a65e5c0f427b5be6d5174a416cc4f203a3cd941 WatchSource:0}: Error finding container 749f03782329967aa6e361b31a65e5c0f427b5be6d5174a416cc4f203a3cd941: Status 404 returned error can't find the container with id 749f03782329967aa6e361b31a65e5c0f427b5be6d5174a416cc4f203a3cd941 Jul 23 15:09:27.580362 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:27.580275 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" event={"ID":"4c88d74d-5ff6-47da-ab89-0ba978b84ee7","Type":"ContainerStarted","Data":"749f03782329967aa6e361b31a65e5c0f427b5be6d5174a416cc4f203a3cd941"} Jul 23 15:09:27.581833 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:27.581799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" event={"ID":"801a2923-1a42-4ba8-8876-4bdf512cfba5","Type":"ContainerStarted","Data":"cf9ceac21eee80884390fdf05c92b2a563cad21e7b26a4012b7de608dd50e0f8"} Jul 23 15:09:27.612821 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:27.612778 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dncdp" podStartSLOduration=1.8052317549999999 podStartE2EDuration="24.612764237s" totalImagesPullingTime="22.807532482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:09:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:09:27.611245623 +0000 UTC m=+555.204696541" watchObservedRunningTime="2026-07-23 15:09:27.612764237 +0000 UTC m=+555.206215164" Jul 23 15:09:31.606287 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:31.606206 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" event={"ID":"4c88d74d-5ff6-47da-ab89-0ba978b84ee7","Type":"ContainerStarted","Data":"5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2"} Jul 23 15:09:31.606633 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:31.606292 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:09:31.631913 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:31.631871 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" podStartSLOduration=1.762349676 podStartE2EDuration="5.631856622s" totalImagesPullingTime="3.869506946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:09:31.630618269 +0000 UTC m=+559.224069219" watchObservedRunningTime="2026-07-23 15:09:31.631856622 +0000 UTC m=+559.225307541" Jul 23 15:09:42.613465 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:09:42.613436 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:10:04.779176 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:04.779146 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rhstc"] Jul 23 15:10:05.043221 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.043138 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhstc" Jul 23 15:10:05.049826 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.049800 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhstc"] Jul 23 15:10:05.050711 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.050690 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-txvgg\"" Jul 23 15:10:05.092637 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.092604 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffd95\" (UniqueName: \"kubernetes.io/projected/3231c792-7306-4153-8971-36cce68a126d-kube-api-access-ffd95\") pod \"authorino-58cd955675-rhstc\" (UID: \"3231c792-7306-4153-8971-36cce68a126d\") " pod="kuadrant-system/authorino-58cd955675-rhstc" Jul 23 15:10:05.193549 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.193513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffd95\" (UniqueName: \"kubernetes.io/projected/3231c792-7306-4153-8971-36cce68a126d-kube-api-access-ffd95\") pod \"authorino-58cd955675-rhstc\" (UID: \"3231c792-7306-4153-8971-36cce68a126d\") " pod="kuadrant-system/authorino-58cd955675-rhstc" Jul 23 15:10:05.205030 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.204982 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffd95\" (UniqueName: \"kubernetes.io/projected/3231c792-7306-4153-8971-36cce68a126d-kube-api-access-ffd95\") pod \"authorino-58cd955675-rhstc\" (UID: \"3231c792-7306-4153-8971-36cce68a126d\") " pod="kuadrant-system/authorino-58cd955675-rhstc" Jul 23 15:10:05.352808 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.352723 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhstc" Jul 23 15:10:05.492849 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.492821 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhstc"] Jul 23 15:10:05.493819 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:10:05.493789 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3231c792_7306_4153_8971_36cce68a126d.slice/crio-eca94b3365f2f8b6fdbcd2533678fa145bb92d1492fbc9e896ab5873fa75cc31 WatchSource:0}: Error finding container eca94b3365f2f8b6fdbcd2533678fa145bb92d1492fbc9e896ab5873fa75cc31: Status 404 returned error can't find the container with id eca94b3365f2f8b6fdbcd2533678fa145bb92d1492fbc9e896ab5873fa75cc31 Jul 23 15:10:05.777870 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:05.777840 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhstc" event={"ID":"3231c792-7306-4153-8971-36cce68a126d","Type":"ContainerStarted","Data":"eca94b3365f2f8b6fdbcd2533678fa145bb92d1492fbc9e896ab5873fa75cc31"} Jul 23 15:10:08.798554 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:08.798518 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhstc" event={"ID":"3231c792-7306-4153-8971-36cce68a126d","Type":"ContainerStarted","Data":"388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4"} Jul 23 15:10:08.824580 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:08.824534 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rhstc" podStartSLOduration=1.6441418410000002 podStartE2EDuration="4.824518915s" totalImagesPullingTime="3.180377074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:10:08.822102342 +0000 UTC m=+596.415553264" watchObservedRunningTime="2026-07-23 15:10:08.824518915 +0000 UTC m=+596.417969833" Jul 23 15:10:11.413492 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.413453 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhstc"] Jul 23 15:10:11.413943 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.413639 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rhstc" podUID="3231c792-7306-4153-8971-36cce68a126d" containerName="authorino" containerID="cri-o://388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4" gracePeriod=30 Jul 23 15:10:11.659383 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.659360 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhstc" Jul 23 15:10:11.751917 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.751888 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffd95\" (UniqueName: \"kubernetes.io/projected/3231c792-7306-4153-8971-36cce68a126d-kube-api-access-ffd95\") pod \"3231c792-7306-4153-8971-36cce68a126d\" (UID: \"3231c792-7306-4153-8971-36cce68a126d\") " Jul 23 15:10:11.753855 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.753828 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3231c792-7306-4153-8971-36cce68a126d-kube-api-access-ffd95" (OuterVolumeSpecName: "kube-api-access-ffd95") pod "3231c792-7306-4153-8971-36cce68a126d" (UID: "3231c792-7306-4153-8971-36cce68a126d"). InnerVolumeSpecName "kube-api-access-ffd95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:10:11.814775 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.814749 2568 generic.go:356] "Generic (PLEG): container finished" podID="3231c792-7306-4153-8971-36cce68a126d" containerID="388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4" exitCode=0 Jul 23 15:10:11.814885 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.814787 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhstc" Jul 23 15:10:11.814885 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.814834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhstc" event={"ID":"3231c792-7306-4153-8971-36cce68a126d","Type":"ContainerDied","Data":"388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4"} Jul 23 15:10:11.814885 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.814869 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhstc" event={"ID":"3231c792-7306-4153-8971-36cce68a126d","Type":"ContainerDied","Data":"eca94b3365f2f8b6fdbcd2533678fa145bb92d1492fbc9e896ab5873fa75cc31"} Jul 23 15:10:11.814885 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.814885 2568 scope.go:117] "RemoveContainer" containerID="388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4" Jul 23 15:10:11.824191 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.824173 2568 scope.go:117] "RemoveContainer" containerID="388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4" Jul 23 15:10:11.824427 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:10:11.824409 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4\": container with ID starting with 388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4 not found: ID does not exist" containerID="388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4" Jul 23 15:10:11.824473 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.824434 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4"} err="failed to get container status \"388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4\": rpc error: code = NotFound desc = could not find container \"388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4\": container with ID starting with 388e876bac6012c221f0486e8e585f77ac727dfaa983a0b65d3d5b750f0198f4 not found: ID does not exist" Jul 23 15:10:11.838429 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.838407 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhstc"] Jul 23 15:10:11.842746 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.842726 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhstc"] Jul 23 15:10:11.852561 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:11.852538 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffd95\" (UniqueName: \"kubernetes.io/projected/3231c792-7306-4153-8971-36cce68a126d-kube-api-access-ffd95\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:10:12.883850 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:12.883818 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:10:12.885023 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:12.885001 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:10:12.887522 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:12.887495 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:10:12.888486 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:12.888464 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:10:12.964965 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:12.964940 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3231c792-7306-4153-8971-36cce68a126d" path="/var/lib/kubelet/pods/3231c792-7306-4153-8971-36cce68a126d/volumes" Jul 23 15:10:40.430955 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.430919 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c4sv"] Jul 23 15:10:40.431433 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.431285 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3231c792-7306-4153-8971-36cce68a126d" containerName="authorino" Jul 23 15:10:40.431433 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.431298 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3231c792-7306-4153-8971-36cce68a126d" containerName="authorino" Jul 23 15:10:40.431433 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.431357 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3231c792-7306-4153-8971-36cce68a126d" containerName="authorino" Jul 23 15:10:40.434341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.434324 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c4sv" Jul 23 15:10:40.437298 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.437279 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-txvgg\"" Jul 23 15:10:40.442536 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.442516 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c4sv"] Jul 23 15:10:40.586372 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.586333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4xl\" (UniqueName: \"kubernetes.io/projected/561cdccc-7051-45d0-9a85-2aaff5267cd0-kube-api-access-td4xl\") pod \"authorino-5d4b67c446-7c4sv\" (UID: \"561cdccc-7051-45d0-9a85-2aaff5267cd0\") " pod="kuadrant-system/authorino-5d4b67c446-7c4sv" Jul 23 15:10:40.687250 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.687170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4xl\" (UniqueName: \"kubernetes.io/projected/561cdccc-7051-45d0-9a85-2aaff5267cd0-kube-api-access-td4xl\") pod \"authorino-5d4b67c446-7c4sv\" (UID: \"561cdccc-7051-45d0-9a85-2aaff5267cd0\") " pod="kuadrant-system/authorino-5d4b67c446-7c4sv" Jul 23 15:10:40.698707 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.698681 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c4sv"] Jul 23 15:10:40.698934 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:10:40.698914 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-td4xl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-7c4sv" podUID="561cdccc-7051-45d0-9a85-2aaff5267cd0" Jul 23 15:10:40.701881 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.701859 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4xl\" (UniqueName: \"kubernetes.io/projected/561cdccc-7051-45d0-9a85-2aaff5267cd0-kube-api-access-td4xl\") pod \"authorino-5d4b67c446-7c4sv\" (UID: \"561cdccc-7051-45d0-9a85-2aaff5267cd0\") " pod="kuadrant-system/authorino-5d4b67c446-7c4sv" Jul 23 15:10:40.730756 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.730728 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-774556ff68-9nrdt"] Jul 23 15:10:40.734209 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.734192 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774556ff68-9nrdt" Jul 23 15:10:40.747536 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.747516 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-774556ff68-9nrdt"] Jul 23 15:10:40.888815 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.888778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p87m\" (UniqueName: \"kubernetes.io/projected/c2ffb0a8-7402-4679-bff7-c2b80b7bacf3-kube-api-access-5p87m\") pod \"authorino-774556ff68-9nrdt\" (UID: \"c2ffb0a8-7402-4679-bff7-c2b80b7bacf3\") " pod="kuadrant-system/authorino-774556ff68-9nrdt" Jul 23 15:10:40.900806 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.900773 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-774556ff68-9nrdt"] Jul 23 15:10:40.901044 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:10:40.901025 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5p87m], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-774556ff68-9nrdt" podUID="c2ffb0a8-7402-4679-bff7-c2b80b7bacf3" Jul 23 15:10:40.968265 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.968240 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c4sv" Jul 23 15:10:40.968416 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.968240 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774556ff68-9nrdt" Jul 23 15:10:40.975261 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.975242 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c4sv" Jul 23 15:10:40.977838 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.977821 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774556ff68-9nrdt" Jul 23 15:10:40.989460 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.989436 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p87m\" (UniqueName: \"kubernetes.io/projected/c2ffb0a8-7402-4679-bff7-c2b80b7bacf3-kube-api-access-5p87m\") pod \"authorino-774556ff68-9nrdt\" (UID: \"c2ffb0a8-7402-4679-bff7-c2b80b7bacf3\") " pod="kuadrant-system/authorino-774556ff68-9nrdt" Jul 23 15:10:40.998286 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:40.998259 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p87m\" (UniqueName: \"kubernetes.io/projected/c2ffb0a8-7402-4679-bff7-c2b80b7bacf3-kube-api-access-5p87m\") pod \"authorino-774556ff68-9nrdt\" (UID: \"c2ffb0a8-7402-4679-bff7-c2b80b7bacf3\") " pod="kuadrant-system/authorino-774556ff68-9nrdt" Jul 23 15:10:41.090372 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.090343 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td4xl\" (UniqueName: \"kubernetes.io/projected/561cdccc-7051-45d0-9a85-2aaff5267cd0-kube-api-access-td4xl\") pod \"561cdccc-7051-45d0-9a85-2aaff5267cd0\" (UID: \"561cdccc-7051-45d0-9a85-2aaff5267cd0\") " Jul 23 15:10:41.090550 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.090475 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p87m\" (UniqueName: \"kubernetes.io/projected/c2ffb0a8-7402-4679-bff7-c2b80b7bacf3-kube-api-access-5p87m\") pod \"c2ffb0a8-7402-4679-bff7-c2b80b7bacf3\" (UID: \"c2ffb0a8-7402-4679-bff7-c2b80b7bacf3\") " Jul 23 15:10:41.092419 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.092390 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ffb0a8-7402-4679-bff7-c2b80b7bacf3-kube-api-access-5p87m" (OuterVolumeSpecName: "kube-api-access-5p87m") pod "c2ffb0a8-7402-4679-bff7-c2b80b7bacf3" (UID: "c2ffb0a8-7402-4679-bff7-c2b80b7bacf3"). InnerVolumeSpecName "kube-api-access-5p87m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:10:41.092515 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.092491 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561cdccc-7051-45d0-9a85-2aaff5267cd0-kube-api-access-td4xl" (OuterVolumeSpecName: "kube-api-access-td4xl") pod "561cdccc-7051-45d0-9a85-2aaff5267cd0" (UID: "561cdccc-7051-45d0-9a85-2aaff5267cd0"). InnerVolumeSpecName "kube-api-access-td4xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:10:41.191426 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.191394 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p87m\" (UniqueName: \"kubernetes.io/projected/c2ffb0a8-7402-4679-bff7-c2b80b7bacf3-kube-api-access-5p87m\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:10:41.191426 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.191423 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td4xl\" (UniqueName: \"kubernetes.io/projected/561cdccc-7051-45d0-9a85-2aaff5267cd0-kube-api-access-td4xl\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:10:41.973268 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.973238 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774556ff68-9nrdt" Jul 23 15:10:41.973663 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:41.973241 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-7c4sv" Jul 23 15:10:42.017925 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:42.017895 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c4sv"] Jul 23 15:10:42.019951 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:42.019926 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-7c4sv"] Jul 23 15:10:42.044344 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:42.044321 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-774556ff68-9nrdt"] Jul 23 15:10:42.048222 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:42.048192 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-774556ff68-9nrdt"] Jul 23 15:10:42.964462 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:42.964428 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="561cdccc-7051-45d0-9a85-2aaff5267cd0" path="/var/lib/kubelet/pods/561cdccc-7051-45d0-9a85-2aaff5267cd0/volumes" Jul 23 15:10:42.964765 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:10:42.964750 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ffb0a8-7402-4679-bff7-c2b80b7bacf3" path="/var/lib/kubelet/pods/c2ffb0a8-7402-4679-bff7-c2b80b7bacf3/volumes" Jul 23 15:12:24.619416 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.619373 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9"] Jul 23 15:12:24.622824 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.622802 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.627144 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.627124 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 15:12:24.627255 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.627166 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f2jgt\"" Jul 23 15:12:24.627324 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.627178 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 23 15:12:24.627324 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.627195 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 15:12:24.635967 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.635946 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9"] Jul 23 15:12:24.690161 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.690135 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.690266 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.690187 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.690266 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.690215 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.690266 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.690243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.690445 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.690281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ec13-2bae-450b-85fe-d453f1fca63b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.690445 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.690301 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxr8\" (UniqueName: \"kubernetes.io/projected/dbc5ec13-2bae-450b-85fe-d453f1fca63b-kube-api-access-5pxr8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.790659 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.790634 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.790767 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.790676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.790767 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.790700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.790767 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.790717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.790767 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.790745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ec13-2bae-450b-85fe-d453f1fca63b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.790767 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.790765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxr8\" (UniqueName: \"kubernetes.io/projected/dbc5ec13-2bae-450b-85fe-d453f1fca63b-kube-api-access-5pxr8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.797438 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.791438 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.797438 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.791639 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.797438 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.791675 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.798165 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.798143 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbc5ec13-2bae-450b-85fe-d453f1fca63b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.798578 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.798558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbc5ec13-2bae-450b-85fe-d453f1fca63b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.806072 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.806047 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxr8\" (UniqueName: \"kubernetes.io/projected/dbc5ec13-2bae-450b-85fe-d453f1fca63b-kube-api-access-5pxr8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9\" (UID: \"dbc5ec13-2bae-450b-85fe-d453f1fca63b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:24.933688 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:24.933633 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:25.073648 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:25.073623 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9"] Jul 23 15:12:25.074591 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:12:25.074568 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc5ec13_2bae_450b_85fe_d453f1fca63b.slice/crio-9e31e8fb35888abacd5ac6794cf98d2a6a657e298abab51066b95dc9ddb11b03 WatchSource:0}: Error finding container 9e31e8fb35888abacd5ac6794cf98d2a6a657e298abab51066b95dc9ddb11b03: Status 404 returned error can't find the container with id 9e31e8fb35888abacd5ac6794cf98d2a6a657e298abab51066b95dc9ddb11b03 Jul 23 15:12:25.076067 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:25.076048 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 15:12:25.482737 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:25.482699 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" event={"ID":"dbc5ec13-2bae-450b-85fe-d453f1fca63b","Type":"ContainerStarted","Data":"9e31e8fb35888abacd5ac6794cf98d2a6a657e298abab51066b95dc9ddb11b03"} Jul 23 15:12:26.489709 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:26.489680 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" event={"ID":"dbc5ec13-2bae-450b-85fe-d453f1fca63b","Type":"ContainerStarted","Data":"fa61b2b61b35154c4bb509f3fea71d8a56d4f6ad0851483acc4a8240cc74df21"} Jul 23 15:12:26.526387 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:26.526336 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" podStartSLOduration=1.4359858939999999 podStartE2EDuration="2.526318649s" totalImagesPullingTime="1.090332755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:12:26.52619204 +0000 UTC m=+734.119642959" watchObservedRunningTime="2026-07-23 15:12:26.526318649 +0000 UTC m=+734.119769567" Jul 23 15:12:34.934680 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:34.934635 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:34.935194 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:34.934713 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:34.947164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:34.947143 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:35.545447 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:35.545423 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9" Jul 23 15:12:46.155150 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.155115 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97"] Jul 23 15:12:46.160045 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.159987 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.166900 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.166884 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 23 15:12:46.181311 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.181288 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97"] Jul 23 15:12:46.235298 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.235270 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.235435 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.235311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7t7d\" (UniqueName: \"kubernetes.io/projected/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-kube-api-access-d7t7d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.235435 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.235333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.235435 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.235407 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.235624 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.235453 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.235624 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.235478 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.335885 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.335849 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7t7d\" (UniqueName: \"kubernetes.io/projected/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-kube-api-access-d7t7d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.335885 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.335886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.336131 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.335930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.336131 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.335967 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.336131 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.336014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.336131 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.336087 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.336404 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.336376 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.336470 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.336389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.336513 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.336466 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.338267 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.338244 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.338539 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.338521 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.346421 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.346397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7t7d\" (UniqueName: \"kubernetes.io/projected/bb597cf9-51f2-4cea-bfb3-118330aa0aa5-kube-api-access-d7t7d\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-86q97\" (UID: \"bb597cf9-51f2-4cea-bfb3-118330aa0aa5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.470077 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.470044 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:46.613541 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:46.613516 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97"] Jul 23 15:12:46.614393 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:12:46.614370 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb597cf9_51f2_4cea_bfb3_118330aa0aa5.slice/crio-4acc553e4af53802074fcc6ab5f18ec2979bdf07a3f29ebdfcb0a7564ee0720c WatchSource:0}: Error finding container 4acc553e4af53802074fcc6ab5f18ec2979bdf07a3f29ebdfcb0a7564ee0720c: Status 404 returned error can't find the container with id 4acc553e4af53802074fcc6ab5f18ec2979bdf07a3f29ebdfcb0a7564ee0720c Jul 23 15:12:47.599341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:47.599301 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" event={"ID":"bb597cf9-51f2-4cea-bfb3-118330aa0aa5","Type":"ContainerStarted","Data":"3c80d445c9841f6f5d3737ed6ec1138d5b8f53e9d6cbe945aa2d2de1cba890df"} Jul 23 15:12:47.599743 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:47.599367 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" event={"ID":"bb597cf9-51f2-4cea-bfb3-118330aa0aa5","Type":"ContainerStarted","Data":"4acc553e4af53802074fcc6ab5f18ec2979bdf07a3f29ebdfcb0a7564ee0720c"} Jul 23 15:12:47.639465 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:47.639414 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" podStartSLOduration=1.40732377 podStartE2EDuration="1.639398832s" totalImagesPullingTime="232.075062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:12:47.637289652 +0000 UTC m=+755.230740572" watchObservedRunningTime="2026-07-23 15:12:47.639398832 +0000 UTC m=+755.232849751" Jul 23 15:12:56.470216 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:56.470175 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:56.470834 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:56.470247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:56.486710 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:56.486683 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:56.658009 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:56.657964 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-86q97" Jul 23 15:12:57.632894 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.632865 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p"] Jul 23 15:12:57.636469 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.636453 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.641146 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.641117 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 23 15:12:57.651958 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.651928 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p"] Jul 23 15:12:57.725141 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.725106 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.725141 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.725142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.725375 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.725169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.725375 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.725254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.725375 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.725290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ddf56-6137-4634-af18-960d86b0eed2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.725484 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.725404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssjg\" (UniqueName: \"kubernetes.io/projected/610ddf56-6137-4634-af18-960d86b0eed2-kube-api-access-bssjg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.826720 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.826688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.826901 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.826744 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.826901 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.826782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ddf56-6137-4634-af18-960d86b0eed2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.826901 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.826807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssjg\" (UniqueName: \"kubernetes.io/projected/610ddf56-6137-4634-af18-960d86b0eed2-kube-api-access-bssjg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.826901 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.826896 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.827159 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.826921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.827218 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.827167 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.827268 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.827214 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.827308 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.827285 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.829068 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.829048 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610ddf56-6137-4634-af18-960d86b0eed2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.829339 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.829321 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610ddf56-6137-4634-af18-960d86b0eed2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.835871 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.835846 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssjg\" (UniqueName: \"kubernetes.io/projected/610ddf56-6137-4634-af18-960d86b0eed2-kube-api-access-bssjg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p\" (UID: \"610ddf56-6137-4634-af18-960d86b0eed2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:57.953326 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:57.953297 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:12:58.090333 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:58.090306 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p"] Jul 23 15:12:58.091176 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:12:58.091150 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod610ddf56_6137_4634_af18_960d86b0eed2.slice/crio-059d2e5bc1b2f5218660ef8920712b3baccb1f2499b89d2fd21a1dcb213691e4 WatchSource:0}: Error finding container 059d2e5bc1b2f5218660ef8920712b3baccb1f2499b89d2fd21a1dcb213691e4: Status 404 returned error can't find the container with id 059d2e5bc1b2f5218660ef8920712b3baccb1f2499b89d2fd21a1dcb213691e4 Jul 23 15:12:58.660001 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:58.659902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" event={"ID":"610ddf56-6137-4634-af18-960d86b0eed2","Type":"ContainerStarted","Data":"002ea4e9acf2f25fc370cf20ac27c8a4802d1660932510657c3023e1ecb84b1a"} Jul 23 15:12:58.660001 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:58.659943 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" event={"ID":"610ddf56-6137-4634-af18-960d86b0eed2","Type":"ContainerStarted","Data":"059d2e5bc1b2f5218660ef8920712b3baccb1f2499b89d2fd21a1dcb213691e4"} Jul 23 15:12:58.681902 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:12:58.681846 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" podStartSLOduration=1.465317359 podStartE2EDuration="1.68183026s" totalImagesPullingTime="216.512901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:12:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:12:58.680424757 +0000 UTC m=+766.273875689" watchObservedRunningTime="2026-07-23 15:12:58.68183026 +0000 UTC m=+766.275281224" Jul 23 15:13:07.954113 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:13:07.954082 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:13:07.954509 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:13:07.954134 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:13:07.966353 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:13:07.966329 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:13:08.731863 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:13:08.731832 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p" Jul 23 15:15:12.919805 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:15:12.919721 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:15:12.923258 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:15:12.923229 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:15:12.924528 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:15:12.924499 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:15:12.928797 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:15:12.928776 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:20:12.960574 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:12.960490 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:20:12.964718 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:12.964694 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:20:12.965331 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:12.965307 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:20:12.969004 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:12.968975 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:20:53.708565 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:53.708528 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm"] Jul 23 15:20:53.711011 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:53.708717 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" podUID="4c88d74d-5ff6-47da-ab89-0ba978b84ee7" containerName="manager" containerID="cri-o://5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2" gracePeriod=10 Jul 23 15:20:56.587073 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:56.587050 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:20:56.649882 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:56.649817 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzcqj\" (UniqueName: \"kubernetes.io/projected/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-kube-api-access-fzcqj\") pod \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " Jul 23 15:20:56.650053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:56.649902 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-extensions-socket-volume\") pod \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\" (UID: \"4c88d74d-5ff6-47da-ab89-0ba978b84ee7\") " Jul 23 15:20:56.650316 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:56.650294 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4c88d74d-5ff6-47da-ab89-0ba978b84ee7" (UID: "4c88d74d-5ff6-47da-ab89-0ba978b84ee7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:20:56.652004 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:56.651964 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-kube-api-access-fzcqj" (OuterVolumeSpecName: "kube-api-access-fzcqj") pod "4c88d74d-5ff6-47da-ab89-0ba978b84ee7" (UID: "4c88d74d-5ff6-47da-ab89-0ba978b84ee7"). InnerVolumeSpecName "kube-api-access-fzcqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:20:56.751479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:56.751455 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcqj\" (UniqueName: \"kubernetes.io/projected/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-kube-api-access-fzcqj\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:20:56.751479 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:56.751477 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c88d74d-5ff6-47da-ab89-0ba978b84ee7-extensions-socket-volume\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:20:57.175732 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.175697 2568 generic.go:356] "Generic (PLEG): container finished" podID="4c88d74d-5ff6-47da-ab89-0ba978b84ee7" containerID="5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2" exitCode=0 Jul 23 15:20:57.175877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.175743 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" Jul 23 15:20:57.175877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.175749 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" event={"ID":"4c88d74d-5ff6-47da-ab89-0ba978b84ee7","Type":"ContainerDied","Data":"5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2"} Jul 23 15:20:57.175877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.175813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm" event={"ID":"4c88d74d-5ff6-47da-ab89-0ba978b84ee7","Type":"ContainerDied","Data":"749f03782329967aa6e361b31a65e5c0f427b5be6d5174a416cc4f203a3cd941"} Jul 23 15:20:57.175877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.175845 2568 scope.go:117] "RemoveContainer" containerID="5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2" Jul 23 15:20:57.186187 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.186170 2568 scope.go:117] "RemoveContainer" containerID="5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2" Jul 23 15:20:57.186437 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:20:57.186392 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2\": container with ID starting with 5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2 not found: ID does not exist" containerID="5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2" Jul 23 15:20:57.186490 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.186437 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2"} err="failed to get container status \"5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2\": rpc error: code = NotFound desc = could not find container \"5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2\": container with ID starting with 5c212085ef45f559f3a76015374aafaa43a8e057bc8f120c10b343ed5a854fa2 not found: ID does not exist" Jul 23 15:20:57.197191 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.197164 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm"] Jul 23 15:20:57.202332 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:57.202314 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g8xpm"] Jul 23 15:20:58.965202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:20:58.965173 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c88d74d-5ff6-47da-ab89-0ba978b84ee7" path="/var/lib/kubelet/pods/4c88d74d-5ff6-47da-ab89-0ba978b84ee7/volumes" Jul 23 15:21:54.930012 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:54.929965 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c"] Jul 23 15:21:54.930470 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:54.930326 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c88d74d-5ff6-47da-ab89-0ba978b84ee7" containerName="manager" Jul 23 15:21:54.930470 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:54.930339 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c88d74d-5ff6-47da-ab89-0ba978b84ee7" containerName="manager" Jul 23 15:21:54.930470 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:54.930415 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c88d74d-5ff6-47da-ab89-0ba978b84ee7" containerName="manager" Jul 23 15:21:54.933313 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:54.933294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:54.940617 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:54.940597 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cddmh\"" Jul 23 15:21:54.948591 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:54.948566 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c"] Jul 23 15:21:55.026975 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.026950 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spwzv\" (UniqueName: \"kubernetes.io/projected/a66e7b80-8997-47e5-85bd-3f9b6061f902-kube-api-access-spwzv\") pod \"kuadrant-operator-controller-manager-86878c769f-4cq4c\" (UID: \"a66e7b80-8997-47e5-85bd-3f9b6061f902\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.027115 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.026987 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a66e7b80-8997-47e5-85bd-3f9b6061f902-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4cq4c\" (UID: \"a66e7b80-8997-47e5-85bd-3f9b6061f902\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.127552 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.127523 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spwzv\" (UniqueName: \"kubernetes.io/projected/a66e7b80-8997-47e5-85bd-3f9b6061f902-kube-api-access-spwzv\") pod \"kuadrant-operator-controller-manager-86878c769f-4cq4c\" (UID: \"a66e7b80-8997-47e5-85bd-3f9b6061f902\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.127673 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.127572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a66e7b80-8997-47e5-85bd-3f9b6061f902-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4cq4c\" (UID: \"a66e7b80-8997-47e5-85bd-3f9b6061f902\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.127948 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.127929 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a66e7b80-8997-47e5-85bd-3f9b6061f902-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4cq4c\" (UID: \"a66e7b80-8997-47e5-85bd-3f9b6061f902\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.137173 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.137150 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spwzv\" (UniqueName: \"kubernetes.io/projected/a66e7b80-8997-47e5-85bd-3f9b6061f902-kube-api-access-spwzv\") pod \"kuadrant-operator-controller-manager-86878c769f-4cq4c\" (UID: \"a66e7b80-8997-47e5-85bd-3f9b6061f902\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.243751 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.243731 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.384978 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.384947 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c"] Jul 23 15:21:55.385668 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:21:55.385645 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66e7b80_8997_47e5_85bd_3f9b6061f902.slice/crio-068925312ce193ac8cb509921359fd055e00236a10c4dc1fac4b941df0831a44 WatchSource:0}: Error finding container 068925312ce193ac8cb509921359fd055e00236a10c4dc1fac4b941df0831a44: Status 404 returned error can't find the container with id 068925312ce193ac8cb509921359fd055e00236a10c4dc1fac4b941df0831a44 Jul 23 15:21:55.474228 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.474202 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" event={"ID":"a66e7b80-8997-47e5-85bd-3f9b6061f902","Type":"ContainerStarted","Data":"8a5c845c61f698850730589b829e9c397e7b7990441912062cb71515fba88cad"} Jul 23 15:21:55.474340 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.474261 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:21:55.474340 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.474284 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" event={"ID":"a66e7b80-8997-47e5-85bd-3f9b6061f902","Type":"ContainerStarted","Data":"068925312ce193ac8cb509921359fd055e00236a10c4dc1fac4b941df0831a44"} Jul 23 15:21:55.499448 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:21:55.499380 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" podStartSLOduration=1.499365342 podStartE2EDuration="1.499365342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:21:55.496285789 +0000 UTC m=+1303.089736709" watchObservedRunningTime="2026-07-23 15:21:55.499365342 +0000 UTC m=+1303.092816260" Jul 23 15:22:06.481737 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:22:06.481711 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cq4c" Jul 23 15:25:13.005611 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:25:13.005582 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:25:13.009250 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:25:13.009222 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:25:13.012230 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:25:13.012204 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:25:13.015655 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:25:13.015637 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:30:13.041686 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:30:13.041657 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:30:13.045302 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:30:13.045282 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:30:13.050114 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:30:13.050097 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:30:13.053456 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:30:13.053437 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:35:13.077588 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:35:13.077556 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:35:13.081534 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:35:13.081513 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:35:13.090501 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:35:13.090485 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:35:13.093942 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:35:13.093925 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:37:43.399136 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.399028 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8"] Jul 23 15:37:43.402451 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.402430 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.406327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.406300 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80ba32\"/\"openshift-service-ca.crt\"" Jul 23 15:37:43.406327 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.406318 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80ba32\"/\"default-dockercfg-7vk45\"" Jul 23 15:37:43.406541 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.406351 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80ba32\"/\"test-model-80ba32-kserve-self-signed-certs\"" Jul 23 15:37:43.406541 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.406407 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80ba32\"/\"kube-root-ca.crt\"" Jul 23 15:37:43.412074 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.412053 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8"] Jul 23 15:37:43.539441 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.539410 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-model-cache\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.539599 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.539460 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1661ca-122a-4c6c-bbfc-220f464c6314-tls-certs\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.539599 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.539513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-tmp-dir\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.539701 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.539595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-home\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.539701 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.539657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-dshm\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.539701 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.539683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzvqn\" (UniqueName: \"kubernetes.io/projected/ff1661ca-122a-4c6c-bbfc-220f464c6314-kube-api-access-kzvqn\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640373 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-home\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640531 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-dshm\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640531 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640432 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzvqn\" (UniqueName: \"kubernetes.io/projected/ff1661ca-122a-4c6c-bbfc-220f464c6314-kube-api-access-kzvqn\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640531 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-model-cache\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640531 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1661ca-122a-4c6c-bbfc-220f464c6314-tls-certs\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640733 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-tmp-dir\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640799 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640779 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-home\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640875 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640856 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-model-cache\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.640927 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.640878 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-tmp-dir\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.642749 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.642718 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-dshm\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.642972 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.642953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1661ca-122a-4c6c-bbfc-220f464c6314-tls-certs\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.649116 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.649097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzvqn\" (UniqueName: \"kubernetes.io/projected/ff1661ca-122a-4c6c-bbfc-220f464c6314-kube-api-access-kzvqn\") pod \"test-model-80ba32-kserve-6ff9557df-r6hn8\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.713139 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.713109 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:43.848843 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:43.848818 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8"] Jul 23 15:37:43.849879 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:37:43.849859 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff1661ca_122a_4c6c_bbfc_220f464c6314.slice/crio-d9233ceb209a758d68e4e5bd042db4ffc702d4a3ad1b97ac68134483e07e9e2b WatchSource:0}: Error finding container d9233ceb209a758d68e4e5bd042db4ffc702d4a3ad1b97ac68134483e07e9e2b: Status 404 returned error can't find the container with id d9233ceb209a758d68e4e5bd042db4ffc702d4a3ad1b97ac68134483e07e9e2b Jul 23 15:37:44.514895 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:44.514863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" event={"ID":"ff1661ca-122a-4c6c-bbfc-220f464c6314","Type":"ContainerStarted","Data":"a1370cb8f1c21ed93e0cd769bd3628caf6550a9b7f48191d3dc110802948bdcf"} Jul 23 15:37:44.515296 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:44.514903 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" event={"ID":"ff1661ca-122a-4c6c-bbfc-220f464c6314","Type":"ContainerStarted","Data":"d9233ceb209a758d68e4e5bd042db4ffc702d4a3ad1b97ac68134483e07e9e2b"} Jul 23 15:37:44.533150 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:44.533102 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" podStartSLOduration=1.533087782 podStartE2EDuration="1.533087782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:37:44.532257788 +0000 UTC m=+2252.125708708" watchObservedRunningTime="2026-07-23 15:37:44.533087782 +0000 UTC m=+2252.126538745" Jul 23 15:37:53.713850 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:53.713816 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:53.714332 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:53.713937 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:53.727384 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:53.727361 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:37:54.576858 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:37:54.576827 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:38:02.606155 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.606103 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89"] Jul 23 15:38:02.609871 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.609846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.614401 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.614376 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-979b59\"/\"test-model-979b59-kserve-self-signed-certs\"" Jul 23 15:38:02.614401 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.614386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-979b59\"/\"default-dockercfg-w27f8\"" Jul 23 15:38:02.614575 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.614376 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-979b59\"/\"openshift-service-ca.crt\"" Jul 23 15:38:02.614736 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.614721 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-979b59\"/\"kube-root-ca.crt\"" Jul 23 15:38:02.630043 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.630019 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89"] Jul 23 15:38:02.794731 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.794697 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-model-cache\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.794904 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.794749 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tmp-dir\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.794904 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.794802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-dshm\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.794904 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.794832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tls-certs\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.795086 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.794909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-home\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.795086 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.794947 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p875w\" (UniqueName: \"kubernetes.io/projected/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-kube-api-access-p875w\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896387 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-model-cache\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896387 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tmp-dir\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896580 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-dshm\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896580 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tls-certs\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896580 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-home\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896580 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p875w\" (UniqueName: \"kubernetes.io/projected/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-kube-api-access-p875w\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896880 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896858 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tmp-dir\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896880 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896873 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-model-cache\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.896986 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.896903 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-home\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.898631 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.898611 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-dshm\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.898956 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.898938 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tls-certs\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.904681 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.904661 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p875w\" (UniqueName: \"kubernetes.io/projected/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-kube-api-access-p875w\") pod \"test-model-979b59-kserve-64748fd995-mvp89\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:02.919634 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:02.919611 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:03.072130 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:03.072105 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89"] Jul 23 15:38:03.073394 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:38:03.073367 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9adf4c49_af2c_4509_b943_c9aa2de7bb4d.slice/crio-113b866e1cc19777f97d7b637c46c039a1618ffb677320e4f1f8f41418be239c WatchSource:0}: Error finding container 113b866e1cc19777f97d7b637c46c039a1618ffb677320e4f1f8f41418be239c: Status 404 returned error can't find the container with id 113b866e1cc19777f97d7b637c46c039a1618ffb677320e4f1f8f41418be239c Jul 23 15:38:03.617023 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:03.616969 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" event={"ID":"9adf4c49-af2c-4509-b943-c9aa2de7bb4d","Type":"ContainerStarted","Data":"22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290"} Jul 23 15:38:03.617416 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:03.617032 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" event={"ID":"9adf4c49-af2c-4509-b943-c9aa2de7bb4d","Type":"ContainerStarted","Data":"113b866e1cc19777f97d7b637c46c039a1618ffb677320e4f1f8f41418be239c"} Jul 23 15:38:03.635185 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:03.635140 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" podStartSLOduration=1.635125885 podStartE2EDuration="1.635125885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:38:03.634146465 +0000 UTC m=+2271.227597378" watchObservedRunningTime="2026-07-23 15:38:03.635125885 +0000 UTC m=+2271.228576833" Jul 23 15:38:12.920378 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:12.920348 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:12.920764 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:12.920427 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:12.932659 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:12.932638 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:13.681043 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:13.681011 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:43.666932 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:43.666897 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8"] Jul 23 15:38:44.572704 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:44.572671 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89"] Jul 23 15:38:58.687651 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:58.687587 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" podUID="ff1661ca-122a-4c6c-bbfc-220f464c6314" containerName="main" containerID="cri-o://a1370cb8f1c21ed93e0cd769bd3628caf6550a9b7f48191d3dc110802948bdcf" gracePeriod=45 Jul 23 15:38:58.913571 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:58.913540 2568 generic.go:356] "Generic (PLEG): container finished" podID="ff1661ca-122a-4c6c-bbfc-220f464c6314" containerID="a1370cb8f1c21ed93e0cd769bd3628caf6550a9b7f48191d3dc110802948bdcf" exitCode=0 Jul 23 15:38:58.913701 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:58.913592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" event={"ID":"ff1661ca-122a-4c6c-bbfc-220f464c6314","Type":"ContainerDied","Data":"a1370cb8f1c21ed93e0cd769bd3628caf6550a9b7f48191d3dc110802948bdcf"} Jul 23 15:38:58.938696 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:58.938645 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:38:59.036696 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.036669 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1661ca-122a-4c6c-bbfc-220f464c6314-tls-certs\") pod \"ff1661ca-122a-4c6c-bbfc-220f464c6314\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " Jul 23 15:38:59.036696 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.036699 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-home\") pod \"ff1661ca-122a-4c6c-bbfc-220f464c6314\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " Jul 23 15:38:59.036863 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.036762 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-tmp-dir\") pod \"ff1661ca-122a-4c6c-bbfc-220f464c6314\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " Jul 23 15:38:59.036863 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.036792 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzvqn\" (UniqueName: \"kubernetes.io/projected/ff1661ca-122a-4c6c-bbfc-220f464c6314-kube-api-access-kzvqn\") pod \"ff1661ca-122a-4c6c-bbfc-220f464c6314\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " Jul 23 15:38:59.036863 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.036833 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-dshm\") pod \"ff1661ca-122a-4c6c-bbfc-220f464c6314\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " Jul 23 15:38:59.036863 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.036858 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-model-cache\") pod \"ff1661ca-122a-4c6c-bbfc-220f464c6314\" (UID: \"ff1661ca-122a-4c6c-bbfc-220f464c6314\") " Jul 23 15:38:59.037096 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.037041 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-home" (OuterVolumeSpecName: "home") pod "ff1661ca-122a-4c6c-bbfc-220f464c6314" (UID: "ff1661ca-122a-4c6c-bbfc-220f464c6314"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.037096 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.037079 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff1661ca-122a-4c6c-bbfc-220f464c6314" (UID: "ff1661ca-122a-4c6c-bbfc-220f464c6314"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.037223 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.037206 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.037276 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.037213 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff1661ca-122a-4c6c-bbfc-220f464c6314" (UID: "ff1661ca-122a-4c6c-bbfc-220f464c6314"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.037276 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.037224 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.038892 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.038864 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1661ca-122a-4c6c-bbfc-220f464c6314-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff1661ca-122a-4c6c-bbfc-220f464c6314" (UID: "ff1661ca-122a-4c6c-bbfc-220f464c6314"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:38:59.039015 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.038897 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-dshm" (OuterVolumeSpecName: "dshm") pod "ff1661ca-122a-4c6c-bbfc-220f464c6314" (UID: "ff1661ca-122a-4c6c-bbfc-220f464c6314"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.039015 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.038966 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1661ca-122a-4c6c-bbfc-220f464c6314-kube-api-access-kzvqn" (OuterVolumeSpecName: "kube-api-access-kzvqn") pod "ff1661ca-122a-4c6c-bbfc-220f464c6314" (UID: "ff1661ca-122a-4c6c-bbfc-220f464c6314"). InnerVolumeSpecName "kube-api-access-kzvqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:38:59.138320 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.138294 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1661ca-122a-4c6c-bbfc-220f464c6314-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.138320 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.138317 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzvqn\" (UniqueName: \"kubernetes.io/projected/ff1661ca-122a-4c6c-bbfc-220f464c6314-kube-api-access-kzvqn\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.138458 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.138327 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.138458 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.138336 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1661ca-122a-4c6c-bbfc-220f464c6314-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.593858 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.593820 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" podUID="9adf4c49-af2c-4509-b943-c9aa2de7bb4d" containerName="main" containerID="cri-o://22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290" gracePeriod=45 Jul 23 15:38:59.846875 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.846819 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:59.920290 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.920263 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" Jul 23 15:38:59.920290 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.920272 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8" event={"ID":"ff1661ca-122a-4c6c-bbfc-220f464c6314","Type":"ContainerDied","Data":"d9233ceb209a758d68e4e5bd042db4ffc702d4a3ad1b97ac68134483e07e9e2b"} Jul 23 15:38:59.920529 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.920322 2568 scope.go:117] "RemoveContainer" containerID="a1370cb8f1c21ed93e0cd769bd3628caf6550a9b7f48191d3dc110802948bdcf" Jul 23 15:38:59.921812 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.921785 2568 generic.go:356] "Generic (PLEG): container finished" podID="9adf4c49-af2c-4509-b943-c9aa2de7bb4d" containerID="22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290" exitCode=0 Jul 23 15:38:59.921959 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.921830 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" Jul 23 15:38:59.921959 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.921866 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" event={"ID":"9adf4c49-af2c-4509-b943-c9aa2de7bb4d","Type":"ContainerDied","Data":"22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290"} Jul 23 15:38:59.921959 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.921899 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89" event={"ID":"9adf4c49-af2c-4509-b943-c9aa2de7bb4d","Type":"ContainerDied","Data":"113b866e1cc19777f97d7b637c46c039a1618ffb677320e4f1f8f41418be239c"} Jul 23 15:38:59.929903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.929887 2568 scope.go:117] "RemoveContainer" containerID="22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290" Jul 23 15:38:59.937867 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.937849 2568 scope.go:117] "RemoveContainer" containerID="22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290" Jul 23 15:38:59.938134 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:38:59.938114 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290\": container with ID starting with 22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290 not found: ID does not exist" containerID="22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290" Jul 23 15:38:59.938226 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.938142 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290"} err="failed to get container status \"22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290\": rpc error: code = NotFound desc = could not find container \"22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290\": container with ID starting with 22574c86c19f13c75607bd61b0b97fda824dcc2d296e9b96625fdb17f7899290 not found: ID does not exist" Jul 23 15:38:59.944161 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944144 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-dshm\") pod \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " Jul 23 15:38:59.944247 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944200 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-home\") pod \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " Jul 23 15:38:59.944247 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944222 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p875w\" (UniqueName: \"kubernetes.io/projected/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-kube-api-access-p875w\") pod \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " Jul 23 15:38:59.944337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944257 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tmp-dir\") pod \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " Jul 23 15:38:59.944337 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944291 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-model-cache\") pod \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " Jul 23 15:38:59.944453 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944338 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tls-certs\") pod \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\" (UID: \"9adf4c49-af2c-4509-b943-c9aa2de7bb4d\") " Jul 23 15:38:59.944509 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944457 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-home" (OuterVolumeSpecName: "home") pod "9adf4c49-af2c-4509-b943-c9aa2de7bb4d" (UID: "9adf4c49-af2c-4509-b943-c9aa2de7bb4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.944566 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944538 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9adf4c49-af2c-4509-b943-c9aa2de7bb4d" (UID: "9adf4c49-af2c-4509-b943-c9aa2de7bb4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.944619 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944606 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9adf4c49-af2c-4509-b943-c9aa2de7bb4d" (UID: "9adf4c49-af2c-4509-b943-c9aa2de7bb4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.944675 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944647 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.944675 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.944666 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:38:59.946264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.946239 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-dshm" (OuterVolumeSpecName: "dshm") pod "9adf4c49-af2c-4509-b943-c9aa2de7bb4d" (UID: "9adf4c49-af2c-4509-b943-c9aa2de7bb4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:38:59.946382 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.946365 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-kube-api-access-p875w" (OuterVolumeSpecName: "kube-api-access-p875w") pod "9adf4c49-af2c-4509-b943-c9aa2de7bb4d" (UID: "9adf4c49-af2c-4509-b943-c9aa2de7bb4d"). InnerVolumeSpecName "kube-api-access-p875w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:38:59.946447 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.946377 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9adf4c49-af2c-4509-b943-c9aa2de7bb4d" (UID: "9adf4c49-af2c-4509-b943-c9aa2de7bb4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:38:59.969948 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.969926 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8"] Jul 23 15:38:59.977793 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:38:59.977773 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-80ba32/test-model-80ba32-kserve-6ff9557df-r6hn8"] Jul 23 15:39:00.045478 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.045453 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:39:00.045616 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.045482 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p875w\" (UniqueName: \"kubernetes.io/projected/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-kube-api-access-p875w\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:39:00.045616 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.045497 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:39:00.045616 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.045511 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9adf4c49-af2c-4509-b943-c9aa2de7bb4d-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:39:00.243715 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.243693 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89"] Jul 23 15:39:00.249905 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.249882 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-979b59/test-model-979b59-kserve-64748fd995-mvp89"] Jul 23 15:39:00.964445 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.964415 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9adf4c49-af2c-4509-b943-c9aa2de7bb4d" path="/var/lib/kubelet/pods/9adf4c49-af2c-4509-b943-c9aa2de7bb4d/volumes" Jul 23 15:39:00.964808 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:39:00.964789 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff1661ca-122a-4c6c-bbfc-220f464c6314" path="/var/lib/kubelet/pods/ff1661ca-122a-4c6c-bbfc-220f464c6314/volumes" Jul 23 15:40:13.113331 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:40:13.113305 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:40:13.116668 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:40:13.116643 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:40:13.126722 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:40:13.126702 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:40:13.129846 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:40:13.129829 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:41:19.890161 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.890115 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb"] Jul 23 15:41:19.890783 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.890662 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9adf4c49-af2c-4509-b943-c9aa2de7bb4d" containerName="main" Jul 23 15:41:19.890783 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.890686 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9adf4c49-af2c-4509-b943-c9aa2de7bb4d" containerName="main" Jul 23 15:41:19.890783 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.890705 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff1661ca-122a-4c6c-bbfc-220f464c6314" containerName="main" Jul 23 15:41:19.890783 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.890714 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff1661ca-122a-4c6c-bbfc-220f464c6314" containerName="main" Jul 23 15:41:19.890957 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.890821 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9adf4c49-af2c-4509-b943-c9aa2de7bb4d" containerName="main" Jul 23 15:41:19.890957 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.890835 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff1661ca-122a-4c6c-bbfc-220f464c6314" containerName="main" Jul 23 15:41:19.894202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.894179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:19.897162 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.897139 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"kube-root-ca.crt\"" Jul 23 15:41:19.898201 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.898175 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"openshift-service-ca.crt\"" Jul 23 15:41:19.898301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.898252 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"auth-test-model-290c54-kserve-self-signed-certs\"" Jul 23 15:41:19.898301 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.898269 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"default-dockercfg-z8gxn\"" Jul 23 15:41:19.905256 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:19.905235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb"] Jul 23 15:41:20.032801 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.032768 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-dshm\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.032971 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.032820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-tmp-dir\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.032971 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.032907 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8nl\" (UniqueName: \"kubernetes.io/projected/385ac53d-d14a-4909-8af0-78a5650d1936-kube-api-access-7m8nl\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.033121 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.033026 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-home\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.033121 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.033067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385ac53d-d14a-4909-8af0-78a5650d1936-tls-certs\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.033193 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.033119 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-model-cache\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.133965 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.133934 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-tmp-dir\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.133981 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8nl\" (UniqueName: \"kubernetes.io/projected/385ac53d-d14a-4909-8af0-78a5650d1936-kube-api-access-7m8nl\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.134034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-home\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.134059 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385ac53d-d14a-4909-8af0-78a5650d1936-tls-certs\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.134091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-model-cache\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134183 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.134126 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-dshm\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134501 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.134372 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-tmp-dir\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134501 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.134441 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-home\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.134584 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.134524 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-model-cache\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.136371 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.136349 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-dshm\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.136701 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.136682 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385ac53d-d14a-4909-8af0-78a5650d1936-tls-certs\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.142219 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.142166 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8nl\" (UniqueName: \"kubernetes.io/projected/385ac53d-d14a-4909-8af0-78a5650d1936-kube-api-access-7m8nl\") pod \"auth-test-model-290c54-kserve-6c7847c987-pjkqb\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.205053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.205029 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:20.339455 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.339428 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb"] Jul 23 15:41:20.341029 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:41:20.340986 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod385ac53d_d14a_4909_8af0_78a5650d1936.slice/crio-c8158786af3bd8ef3c9c255b4107dbc4fe0569e21c5c7c2d4357b658c585e0b3 WatchSource:0}: Error finding container c8158786af3bd8ef3c9c255b4107dbc4fe0569e21c5c7c2d4357b658c585e0b3: Status 404 returned error can't find the container with id c8158786af3bd8ef3c9c255b4107dbc4fe0569e21c5c7c2d4357b658c585e0b3 Jul 23 15:41:20.679159 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.679066 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" event={"ID":"385ac53d-d14a-4909-8af0-78a5650d1936","Type":"ContainerStarted","Data":"395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2"} Jul 23 15:41:20.679159 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.679104 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" event={"ID":"385ac53d-d14a-4909-8af0-78a5650d1936","Type":"ContainerStarted","Data":"c8158786af3bd8ef3c9c255b4107dbc4fe0569e21c5c7c2d4357b658c585e0b3"} Jul 23 15:41:20.699009 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:20.698941 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" podStartSLOduration=1.698926797 podStartE2EDuration="1.698926797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:41:20.698712484 +0000 UTC m=+2468.292163407" watchObservedRunningTime="2026-07-23 15:41:20.698926797 +0000 UTC m=+2468.292377745" Jul 23 15:41:30.205857 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:30.205821 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:30.206369 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:30.205875 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:30.218273 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:30.218246 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:30.742451 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:30.742420 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:41:54.319640 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.319603 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4"] Jul 23 15:41:54.323808 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.323787 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.326821 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.326795 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"openshift-service-ca.crt\"" Jul 23 15:41:54.326938 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.326818 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"default-dockercfg-pkvfj\"" Jul 23 15:41:54.326938 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.326845 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"auth-test-model-d05e2c-kserve-self-signed-certs\"" Jul 23 15:41:54.327089 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.326941 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"kube-root-ca.crt\"" Jul 23 15:41:54.333545 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.333523 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4"] Jul 23 15:41:54.420048 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.420016 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz4k4\" (UniqueName: \"kubernetes.io/projected/2cf673c0-e90b-4e99-a9ff-59b579893e8b-kube-api-access-zz4k4\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.420197 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.420054 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-home\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.420197 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.420138 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tls-certs\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.420197 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.420178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-dshm\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.420316 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.420218 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tmp-dir\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.420316 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.420241 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-model-cache\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521370 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521336 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-home\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521551 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521402 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tls-certs\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521551 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-dshm\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521551 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tmp-dir\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521551 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-model-cache\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521718 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521588 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz4k4\" (UniqueName: \"kubernetes.io/projected/2cf673c0-e90b-4e99-a9ff-59b579893e8b-kube-api-access-zz4k4\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521834 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-home\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521922 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tmp-dir\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.521972 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.521937 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-model-cache\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.523792 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.523773 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-dshm\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.523988 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.523973 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tls-certs\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.530541 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.530516 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz4k4\" (UniqueName: \"kubernetes.io/projected/2cf673c0-e90b-4e99-a9ff-59b579893e8b-kube-api-access-zz4k4\") pod \"auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.635428 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.635359 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:41:54.769264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.769238 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4"] Jul 23 15:41:54.771420 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:41:54.771393 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cf673c0_e90b_4e99_a9ff_59b579893e8b.slice/crio-09d48f9a67b53c18e9b5db49f4cb412030b1d66588527bbe7c8bd5785c3d057e WatchSource:0}: Error finding container 09d48f9a67b53c18e9b5db49f4cb412030b1d66588527bbe7c8bd5785c3d057e: Status 404 returned error can't find the container with id 09d48f9a67b53c18e9b5db49f4cb412030b1d66588527bbe7c8bd5785c3d057e Jul 23 15:41:54.858971 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.858941 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" event={"ID":"2cf673c0-e90b-4e99-a9ff-59b579893e8b","Type":"ContainerStarted","Data":"ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e"} Jul 23 15:41:54.859120 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.859005 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" event={"ID":"2cf673c0-e90b-4e99-a9ff-59b579893e8b","Type":"ContainerStarted","Data":"09d48f9a67b53c18e9b5db49f4cb412030b1d66588527bbe7c8bd5785c3d057e"} Jul 23 15:41:54.877648 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:41:54.877595 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" podStartSLOduration=0.877577601 podStartE2EDuration="877.577601ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:41:54.877081069 +0000 UTC m=+2502.470531989" watchObservedRunningTime="2026-07-23 15:41:54.877577601 +0000 UTC m=+2502.471028521" Jul 23 15:42:04.635923 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:42:04.635889 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:42:04.635923 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:42:04.635932 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:42:04.648954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:42:04.648924 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:42:04.924177 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:42:04.924077 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:44:07.821585 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:07.821508 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb"] Jul 23 15:44:08.706403 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:08.706373 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4"] Jul 23 15:44:17.399902 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.399867 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m"] Jul 23 15:44:17.403498 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.403468 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.406323 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.406302 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"sub-test-model-290c54-kserve-self-signed-certs\"" Jul 23 15:44:17.414882 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.414860 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m"] Jul 23 15:44:17.503202 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.503172 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf62t\" (UniqueName: \"kubernetes.io/projected/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-kube-api-access-lf62t\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.503376 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.503231 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-model-cache\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.503376 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.503257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tmp-dir\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.503376 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.503283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tls-certs\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.503376 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.503343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-home\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.503376 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.503368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-dshm\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.603795 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.603759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf62t\" (UniqueName: \"kubernetes.io/projected/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-kube-api-access-lf62t\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.603951 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.603817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-model-cache\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.603951 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.603921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tmp-dir\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.604065 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.603968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tls-certs\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.604115 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.604079 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-home\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.604162 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.604118 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-dshm\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.604199 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.604173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-model-cache\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.604323 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.604303 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tmp-dir\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.604401 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.604382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-home\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.606233 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.606205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-dshm\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.606502 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.606484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tls-certs\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.613172 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.613149 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf62t\" (UniqueName: \"kubernetes.io/projected/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-kube-api-access-lf62t\") pod \"sub-test-model-290c54-kserve-65c866c77-mzm5m\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.712999 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.712968 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:17.846287 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:17.846263 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m"] Jul 23 15:44:17.848485 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:44:17.848460 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e7be690_e4f8_448c_8c8e_1656f5ccbc31.slice/crio-c0dcb17737f228bf417f67bf689cdbfb3fdd13328f330883130719c68e2cf278 WatchSource:0}: Error finding container c0dcb17737f228bf417f67bf689cdbfb3fdd13328f330883130719c68e2cf278: Status 404 returned error can't find the container with id c0dcb17737f228bf417f67bf689cdbfb3fdd13328f330883130719c68e2cf278 Jul 23 15:44:18.626420 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:18.626380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" event={"ID":"2e7be690-e4f8-448c-8c8e-1656f5ccbc31","Type":"ContainerStarted","Data":"1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34"} Jul 23 15:44:18.626839 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:18.626425 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" event={"ID":"2e7be690-e4f8-448c-8c8e-1656f5ccbc31","Type":"ContainerStarted","Data":"c0dcb17737f228bf417f67bf689cdbfb3fdd13328f330883130719c68e2cf278"} Jul 23 15:44:18.646438 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:18.646390 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" podStartSLOduration=1.646375535 podStartE2EDuration="1.646375535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:44:18.644129428 +0000 UTC m=+2646.237580358" watchObservedRunningTime="2026-07-23 15:44:18.646375535 +0000 UTC m=+2646.239826467" Jul 23 15:44:22.841738 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:22.841681 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" podUID="385ac53d-d14a-4909-8af0-78a5650d1936" containerName="main" containerID="cri-o://395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2" gracePeriod=45 Jul 23 15:44:23.101053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.100987 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:44:23.249204 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249177 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-model-cache\") pod \"385ac53d-d14a-4909-8af0-78a5650d1936\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " Jul 23 15:44:23.249374 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249216 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-tmp-dir\") pod \"385ac53d-d14a-4909-8af0-78a5650d1936\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " Jul 23 15:44:23.249374 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249245 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7m8nl\" (UniqueName: \"kubernetes.io/projected/385ac53d-d14a-4909-8af0-78a5650d1936-kube-api-access-7m8nl\") pod \"385ac53d-d14a-4909-8af0-78a5650d1936\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " Jul 23 15:44:23.249374 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249307 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-home\") pod \"385ac53d-d14a-4909-8af0-78a5650d1936\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " Jul 23 15:44:23.249374 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249344 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385ac53d-d14a-4909-8af0-78a5650d1936-tls-certs\") pod \"385ac53d-d14a-4909-8af0-78a5650d1936\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " Jul 23 15:44:23.249583 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249407 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-dshm\") pod \"385ac53d-d14a-4909-8af0-78a5650d1936\" (UID: \"385ac53d-d14a-4909-8af0-78a5650d1936\") " Jul 23 15:44:23.249583 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249473 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-model-cache" (OuterVolumeSpecName: "model-cache") pod "385ac53d-d14a-4909-8af0-78a5650d1936" (UID: "385ac53d-d14a-4909-8af0-78a5650d1936"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:23.249583 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249529 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "385ac53d-d14a-4909-8af0-78a5650d1936" (UID: "385ac53d-d14a-4909-8af0-78a5650d1936"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:23.249583 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249572 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-home" (OuterVolumeSpecName: "home") pod "385ac53d-d14a-4909-8af0-78a5650d1936" (UID: "385ac53d-d14a-4909-8af0-78a5650d1936"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:23.249795 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249718 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:23.249795 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249737 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:23.249795 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.249749 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:23.251845 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.251821 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385ac53d-d14a-4909-8af0-78a5650d1936-kube-api-access-7m8nl" (OuterVolumeSpecName: "kube-api-access-7m8nl") pod "385ac53d-d14a-4909-8af0-78a5650d1936" (UID: "385ac53d-d14a-4909-8af0-78a5650d1936"). InnerVolumeSpecName "kube-api-access-7m8nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:44:23.251845 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.251829 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-dshm" (OuterVolumeSpecName: "dshm") pod "385ac53d-d14a-4909-8af0-78a5650d1936" (UID: "385ac53d-d14a-4909-8af0-78a5650d1936"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:23.252023 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.251897 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385ac53d-d14a-4909-8af0-78a5650d1936-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "385ac53d-d14a-4909-8af0-78a5650d1936" (UID: "385ac53d-d14a-4909-8af0-78a5650d1936"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:44:23.351154 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.351087 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m8nl\" (UniqueName: \"kubernetes.io/projected/385ac53d-d14a-4909-8af0-78a5650d1936-kube-api-access-7m8nl\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:23.351154 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.351110 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385ac53d-d14a-4909-8af0-78a5650d1936-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:23.351154 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.351122 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385ac53d-d14a-4909-8af0-78a5650d1936-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:23.654271 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.654192 2568 generic.go:356] "Generic (PLEG): container finished" podID="385ac53d-d14a-4909-8af0-78a5650d1936" containerID="395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2" exitCode=0 Jul 23 15:44:23.654271 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.654232 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" Jul 23 15:44:23.654440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.654271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" event={"ID":"385ac53d-d14a-4909-8af0-78a5650d1936","Type":"ContainerDied","Data":"395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2"} Jul 23 15:44:23.654440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.654313 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb" event={"ID":"385ac53d-d14a-4909-8af0-78a5650d1936","Type":"ContainerDied","Data":"c8158786af3bd8ef3c9c255b4107dbc4fe0569e21c5c7c2d4357b658c585e0b3"} Jul 23 15:44:23.654440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.654330 2568 scope.go:117] "RemoveContainer" containerID="395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2" Jul 23 15:44:23.663720 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.663705 2568 scope.go:117] "RemoveContainer" containerID="395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2" Jul 23 15:44:23.663945 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:44:23.663926 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2\": container with ID starting with 395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2 not found: ID does not exist" containerID="395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2" Jul 23 15:44:23.664030 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.663956 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2"} err="failed to get container status \"395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2\": rpc error: code = NotFound desc = could not find container \"395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2\": container with ID starting with 395866f52e02694c82ac9681731d3cfd080772c65e5ce8b8b28a570838b4adb2 not found: ID does not exist" Jul 23 15:44:23.676259 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.676236 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb"] Jul 23 15:44:23.680835 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.680815 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/auth-test-model-290c54-kserve-6c7847c987-pjkqb"] Jul 23 15:44:23.726164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.726138 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" podUID="2cf673c0-e90b-4e99-a9ff-59b579893e8b" containerName="main" containerID="cri-o://ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e" gracePeriod=45 Jul 23 15:44:23.980754 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:23.980727 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:44:24.055936 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.055846 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tmp-dir\") pod \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " Jul 23 15:44:24.056089 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.055982 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-home\") pod \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " Jul 23 15:44:24.056089 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056020 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2cf673c0-e90b-4e99-a9ff-59b579893e8b" (UID: "2cf673c0-e90b-4e99-a9ff-59b579893e8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:24.056089 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056076 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-dshm\") pod \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " Jul 23 15:44:24.056280 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056119 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-model-cache\") pod \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " Jul 23 15:44:24.056280 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056151 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zz4k4\" (UniqueName: \"kubernetes.io/projected/2cf673c0-e90b-4e99-a9ff-59b579893e8b-kube-api-access-zz4k4\") pod \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " Jul 23 15:44:24.056280 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056191 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tls-certs\") pod \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\" (UID: \"2cf673c0-e90b-4e99-a9ff-59b579893e8b\") " Jul 23 15:44:24.056280 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056212 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-home" (OuterVolumeSpecName: "home") pod "2cf673c0-e90b-4e99-a9ff-59b579893e8b" (UID: "2cf673c0-e90b-4e99-a9ff-59b579893e8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:24.056492 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056435 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2cf673c0-e90b-4e99-a9ff-59b579893e8b" (UID: "2cf673c0-e90b-4e99-a9ff-59b579893e8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:24.056492 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056461 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:24.056492 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.056481 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:24.058257 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.058233 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-dshm" (OuterVolumeSpecName: "dshm") pod "2cf673c0-e90b-4e99-a9ff-59b579893e8b" (UID: "2cf673c0-e90b-4e99-a9ff-59b579893e8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:44:24.058361 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.058292 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cf673c0-e90b-4e99-a9ff-59b579893e8b" (UID: "2cf673c0-e90b-4e99-a9ff-59b579893e8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:44:24.058361 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.058308 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf673c0-e90b-4e99-a9ff-59b579893e8b-kube-api-access-zz4k4" (OuterVolumeSpecName: "kube-api-access-zz4k4") pod "2cf673c0-e90b-4e99-a9ff-59b579893e8b" (UID: "2cf673c0-e90b-4e99-a9ff-59b579893e8b"). InnerVolumeSpecName "kube-api-access-zz4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:44:24.157780 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.157759 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:24.157780 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.157778 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf673c0-e90b-4e99-a9ff-59b579893e8b-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:24.157916 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.157788 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz4k4\" (UniqueName: \"kubernetes.io/projected/2cf673c0-e90b-4e99-a9ff-59b579893e8b-kube-api-access-zz4k4\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:24.157916 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.157797 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf673c0-e90b-4e99-a9ff-59b579893e8b-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:44:24.661447 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.661414 2568 generic.go:356] "Generic (PLEG): container finished" podID="2cf673c0-e90b-4e99-a9ff-59b579893e8b" containerID="ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e" exitCode=0 Jul 23 15:44:24.661638 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.661456 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" Jul 23 15:44:24.661638 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.661500 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" event={"ID":"2cf673c0-e90b-4e99-a9ff-59b579893e8b","Type":"ContainerDied","Data":"ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e"} Jul 23 15:44:24.661638 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.661546 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4" event={"ID":"2cf673c0-e90b-4e99-a9ff-59b579893e8b","Type":"ContainerDied","Data":"09d48f9a67b53c18e9b5db49f4cb412030b1d66588527bbe7c8bd5785c3d057e"} Jul 23 15:44:24.661638 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.661569 2568 scope.go:117] "RemoveContainer" containerID="ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e" Jul 23 15:44:24.671483 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.671469 2568 scope.go:117] "RemoveContainer" containerID="ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e" Jul 23 15:44:24.671743 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:44:24.671726 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e\": container with ID starting with ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e not found: ID does not exist" containerID="ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e" Jul 23 15:44:24.671799 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.671753 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e"} err="failed to get container status \"ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e\": rpc error: code = NotFound desc = could not find container \"ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e\": container with ID starting with ca7bf5ba37a3455a761e8db6006bf4f3cb164ce0b2479b8eb345c74a1cc7bb8e not found: ID does not exist" Jul 23 15:44:24.684948 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.684926 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4"] Jul 23 15:44:24.696211 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.691561 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/auth-test-model-d05e2c-kserve-6d58585f5f-8z5s4"] Jul 23 15:44:24.965146 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.965116 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf673c0-e90b-4e99-a9ff-59b579893e8b" path="/var/lib/kubelet/pods/2cf673c0-e90b-4e99-a9ff-59b579893e8b/volumes" Jul 23 15:44:24.965502 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:24.965488 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385ac53d-d14a-4909-8af0-78a5650d1936" path="/var/lib/kubelet/pods/385ac53d-d14a-4909-8af0-78a5650d1936/volumes" Jul 23 15:44:27.713900 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:27.713862 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:27.714324 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:27.713916 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:27.726379 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:27.726352 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:44:28.697380 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:44:28.697354 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:45:13.147711 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:13.147639 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:45:13.151368 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:13.151347 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:45:13.164496 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:13.164476 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:45:13.168063 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:13.168042 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-66ftl_eecc2928-3d95-4b0b-8d39-2777137e21bf/ovn-acl-logging/0.log" Jul 23 15:45:23.514399 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.514369 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh"] Jul 23 15:45:23.514949 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.514886 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385ac53d-d14a-4909-8af0-78a5650d1936" containerName="main" Jul 23 15:45:23.514949 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.514908 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="385ac53d-d14a-4909-8af0-78a5650d1936" containerName="main" Jul 23 15:45:23.514949 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.514922 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf673c0-e90b-4e99-a9ff-59b579893e8b" containerName="main" Jul 23 15:45:23.514949 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.514931 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf673c0-e90b-4e99-a9ff-59b579893e8b" containerName="main" Jul 23 15:45:23.515195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.515082 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf673c0-e90b-4e99-a9ff-59b579893e8b" containerName="main" Jul 23 15:45:23.515195 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.515100 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="385ac53d-d14a-4909-8af0-78a5650d1936" containerName="main" Jul 23 15:45:23.518120 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.518099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.520860 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.520837 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"openshift-service-ca.crt\"" Jul 23 15:45:23.521926 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.521905 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"sub-test-model-d05e2c-kserve-self-signed-certs\"" Jul 23 15:45:23.522060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.521932 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"default-dockercfg-pkvfj\"" Jul 23 15:45:23.522060 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.521951 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"kube-root-ca.crt\"" Jul 23 15:45:23.529177 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.529152 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh"] Jul 23 15:45:23.625170 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.625147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tls-certs\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.625283 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.625184 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-model-cache\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.625283 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.625211 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tmp-dir\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.625359 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.625305 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncv8s\" (UniqueName: \"kubernetes.io/projected/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-kube-api-access-ncv8s\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.625398 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.625358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-home\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.625398 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.625381 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-dshm\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726063 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726040 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-model-cache\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tmp-dir\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726106 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncv8s\" (UniqueName: \"kubernetes.io/projected/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-kube-api-access-ncv8s\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-home\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726162 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-dshm\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tls-certs\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726494 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726426 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-model-cache\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726550 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tmp-dir\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.726550 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.726532 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-home\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.728478 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.728453 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-dshm\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.728594 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.728578 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tls-certs\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.734869 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.734849 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncv8s\" (UniqueName: \"kubernetes.io/projected/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-kube-api-access-ncv8s\") pod \"sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.828743 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.828688 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:23.969422 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.969393 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh"] Jul 23 15:45:23.970700 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:45:23.970669 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1740b768_4fc9_4e52_b6dc_58b8b83b6e84.slice/crio-dd1edb1b4a3f114ffedcb8697b3d753c1afd9ca1b05af603b2b713bc413314e1 WatchSource:0}: Error finding container dd1edb1b4a3f114ffedcb8697b3d753c1afd9ca1b05af603b2b713bc413314e1: Status 404 returned error can't find the container with id dd1edb1b4a3f114ffedcb8697b3d753c1afd9ca1b05af603b2b713bc413314e1 Jul 23 15:45:23.982269 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:23.982246 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" event={"ID":"1740b768-4fc9-4e52-b6dc-58b8b83b6e84","Type":"ContainerStarted","Data":"dd1edb1b4a3f114ffedcb8697b3d753c1afd9ca1b05af603b2b713bc413314e1"} Jul 23 15:45:24.989224 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:24.989191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" event={"ID":"1740b768-4fc9-4e52-b6dc-58b8b83b6e84","Type":"ContainerStarted","Data":"72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d"} Jul 23 15:45:25.012378 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:25.012330 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" podStartSLOduration=2.01231509 podStartE2EDuration="2.01231509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:45:25.009688276 +0000 UTC m=+2712.603139192" watchObservedRunningTime="2026-07-23 15:45:25.01231509 +0000 UTC m=+2712.605766009" Jul 23 15:45:33.829387 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:33.829347 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:33.829875 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:33.829404 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:33.841820 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:33.841795 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:45:34.047574 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:45:34.047550 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:46:05.969840 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:05.969807 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m"] Jul 23 15:46:06.812484 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:06.812455 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh"] Jul 23 15:46:20.989793 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:20.989752 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" podUID="2e7be690-e4f8-448c-8c8e-1656f5ccbc31" containerName="main" containerID="cri-o://1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34" gracePeriod=45 Jul 23 15:46:21.245027 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.244937 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:46:21.293706 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.293670 2568 generic.go:356] "Generic (PLEG): container finished" podID="2e7be690-e4f8-448c-8c8e-1656f5ccbc31" containerID="1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34" exitCode=0 Jul 23 15:46:21.293873 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.293711 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" Jul 23 15:46:21.293873 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.293721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" event={"ID":"2e7be690-e4f8-448c-8c8e-1656f5ccbc31","Type":"ContainerDied","Data":"1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34"} Jul 23 15:46:21.293873 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.293760 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m" event={"ID":"2e7be690-e4f8-448c-8c8e-1656f5ccbc31","Type":"ContainerDied","Data":"c0dcb17737f228bf417f67bf689cdbfb3fdd13328f330883130719c68e2cf278"} Jul 23 15:46:21.293873 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.293783 2568 scope.go:117] "RemoveContainer" containerID="1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34" Jul 23 15:46:21.302566 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.302545 2568 scope.go:117] "RemoveContainer" containerID="1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34" Jul 23 15:46:21.302818 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:46:21.302799 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34\": container with ID starting with 1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34 not found: ID does not exist" containerID="1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34" Jul 23 15:46:21.302882 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.302832 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34"} err="failed to get container status \"1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34\": rpc error: code = NotFound desc = could not find container \"1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34\": container with ID starting with 1234ad203fa3009185da3b2d0bfd4a63308ab08d24147776e238bba5d5031e34 not found: ID does not exist" Jul 23 15:46:21.314177 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314152 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-home\") pod \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " Jul 23 15:46:21.314266 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314241 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-dshm\") pod \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " Jul 23 15:46:21.314314 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314284 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf62t\" (UniqueName: \"kubernetes.io/projected/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-kube-api-access-lf62t\") pod \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " Jul 23 15:46:21.314374 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314316 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tmp-dir\") pod \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " Jul 23 15:46:21.314374 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314348 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tls-certs\") pod \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " Jul 23 15:46:21.314475 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314375 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-model-cache\") pod \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\" (UID: \"2e7be690-e4f8-448c-8c8e-1656f5ccbc31\") " Jul 23 15:46:21.314475 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314432 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-home" (OuterVolumeSpecName: "home") pod "2e7be690-e4f8-448c-8c8e-1656f5ccbc31" (UID: "2e7be690-e4f8-448c-8c8e-1656f5ccbc31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:21.314666 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314620 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2e7be690-e4f8-448c-8c8e-1656f5ccbc31" (UID: "2e7be690-e4f8-448c-8c8e-1656f5ccbc31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:21.314782 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314683 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:21.314782 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.314687 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e7be690-e4f8-448c-8c8e-1656f5ccbc31" (UID: "2e7be690-e4f8-448c-8c8e-1656f5ccbc31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:21.316527 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.316502 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-kube-api-access-lf62t" (OuterVolumeSpecName: "kube-api-access-lf62t") pod "2e7be690-e4f8-448c-8c8e-1656f5ccbc31" (UID: "2e7be690-e4f8-448c-8c8e-1656f5ccbc31"). InnerVolumeSpecName "kube-api-access-lf62t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:46:21.316527 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.316509 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-dshm" (OuterVolumeSpecName: "dshm") pod "2e7be690-e4f8-448c-8c8e-1656f5ccbc31" (UID: "2e7be690-e4f8-448c-8c8e-1656f5ccbc31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:21.316660 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.316587 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e7be690-e4f8-448c-8c8e-1656f5ccbc31" (UID: "2e7be690-e4f8-448c-8c8e-1656f5ccbc31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:46:21.415888 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.415865 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:21.415888 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.415886 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf62t\" (UniqueName: \"kubernetes.io/projected/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-kube-api-access-lf62t\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:21.416067 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.415897 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:21.416067 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.415908 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:21.416067 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.415916 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7be690-e4f8-448c-8c8e-1656f5ccbc31-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:21.617186 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.617152 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m"] Jul 23 15:46:21.619487 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.619466 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/sub-test-model-290c54-kserve-65c866c77-mzm5m"] Jul 23 15:46:21.832788 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:21.832734 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" podUID="1740b768-4fc9-4e52-b6dc-58b8b83b6e84" containerName="main" containerID="cri-o://72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d" gracePeriod=45 Jul 23 15:46:22.092602 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.092581 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:46:22.221209 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221138 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-dshm\") pod \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " Jul 23 15:46:22.221209 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221171 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tls-certs\") pod \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " Jul 23 15:46:22.221209 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221210 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tmp-dir\") pod \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " Jul 23 15:46:22.221469 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221240 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-model-cache\") pod \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " Jul 23 15:46:22.221469 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221270 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncv8s\" (UniqueName: \"kubernetes.io/projected/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-kube-api-access-ncv8s\") pod \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " Jul 23 15:46:22.221469 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221285 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-home\") pod \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\" (UID: \"1740b768-4fc9-4e52-b6dc-58b8b83b6e84\") " Jul 23 15:46:22.221641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221483 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-model-cache" (OuterVolumeSpecName: "model-cache") pod "1740b768-4fc9-4e52-b6dc-58b8b83b6e84" (UID: "1740b768-4fc9-4e52-b6dc-58b8b83b6e84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:22.221641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221550 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1740b768-4fc9-4e52-b6dc-58b8b83b6e84" (UID: "1740b768-4fc9-4e52-b6dc-58b8b83b6e84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:22.221641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221590 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:22.221641 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.221588 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-home" (OuterVolumeSpecName: "home") pod "1740b768-4fc9-4e52-b6dc-58b8b83b6e84" (UID: "1740b768-4fc9-4e52-b6dc-58b8b83b6e84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:22.223208 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.223180 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1740b768-4fc9-4e52-b6dc-58b8b83b6e84" (UID: "1740b768-4fc9-4e52-b6dc-58b8b83b6e84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:46:22.223310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.223202 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-dshm" (OuterVolumeSpecName: "dshm") pod "1740b768-4fc9-4e52-b6dc-58b8b83b6e84" (UID: "1740b768-4fc9-4e52-b6dc-58b8b83b6e84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:46:22.223310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.223289 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-kube-api-access-ncv8s" (OuterVolumeSpecName: "kube-api-access-ncv8s") pod "1740b768-4fc9-4e52-b6dc-58b8b83b6e84" (UID: "1740b768-4fc9-4e52-b6dc-58b8b83b6e84"). InnerVolumeSpecName "kube-api-access-ncv8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:46:22.300607 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.300580 2568 generic.go:356] "Generic (PLEG): container finished" podID="1740b768-4fc9-4e52-b6dc-58b8b83b6e84" containerID="72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d" exitCode=0 Jul 23 15:46:22.300724 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.300620 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" Jul 23 15:46:22.300724 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.300643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" event={"ID":"1740b768-4fc9-4e52-b6dc-58b8b83b6e84","Type":"ContainerDied","Data":"72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d"} Jul 23 15:46:22.300724 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.300681 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh" event={"ID":"1740b768-4fc9-4e52-b6dc-58b8b83b6e84","Type":"ContainerDied","Data":"dd1edb1b4a3f114ffedcb8697b3d753c1afd9ca1b05af603b2b713bc413314e1"} Jul 23 15:46:22.300724 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.300698 2568 scope.go:117] "RemoveContainer" containerID="72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d" Jul 23 15:46:22.309835 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.309817 2568 scope.go:117] "RemoveContainer" containerID="72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d" Jul 23 15:46:22.310120 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:46:22.310103 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d\": container with ID starting with 72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d not found: ID does not exist" containerID="72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d" Jul 23 15:46:22.310173 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.310127 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d"} err="failed to get container status \"72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d\": rpc error: code = NotFound desc = could not find container \"72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d\": container with ID starting with 72fcdd615feaaed17ae167570fc593b40ef6eb8df43ae90af905ad7d5c70897d not found: ID does not exist" Jul 23 15:46:22.322461 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.322442 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:22.322461 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.322462 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:22.322586 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.322470 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:22.322586 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.322478 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncv8s\" (UniqueName: \"kubernetes.io/projected/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-kube-api-access-ncv8s\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:22.322586 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.322488 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1740b768-4fc9-4e52-b6dc-58b8b83b6e84-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:46:22.323900 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.323878 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh"] Jul 23 15:46:22.325959 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.325938 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/sub-test-model-d05e2c-kserve-54fb9659fc-xfhbh"] Jul 23 15:46:22.964983 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.964951 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1740b768-4fc9-4e52-b6dc-58b8b83b6e84" path="/var/lib/kubelet/pods/1740b768-4fc9-4e52-b6dc-58b8b83b6e84/volumes" Jul 23 15:46:22.965495 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:22.965474 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7be690-e4f8-448c-8c8e-1656f5ccbc31" path="/var/lib/kubelet/pods/2e7be690-e4f8-448c-8c8e-1656f5ccbc31/volumes" Jul 23 15:46:27.811620 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.811588 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp"] Jul 23 15:46:27.812213 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.812187 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7be690-e4f8-448c-8c8e-1656f5ccbc31" containerName="main" Jul 23 15:46:27.812297 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.812217 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7be690-e4f8-448c-8c8e-1656f5ccbc31" containerName="main" Jul 23 15:46:27.812297 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.812240 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1740b768-4fc9-4e52-b6dc-58b8b83b6e84" containerName="main" Jul 23 15:46:27.812297 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.812249 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1740b768-4fc9-4e52-b6dc-58b8b83b6e84" containerName="main" Jul 23 15:46:27.812452 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.812355 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1740b768-4fc9-4e52-b6dc-58b8b83b6e84" containerName="main" Jul 23 15:46:27.812452 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.812371 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7be690-e4f8-448c-8c8e-1656f5ccbc31" containerName="main" Jul 23 15:46:27.816739 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.816715 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.820498 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.820467 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"openshift-service-ca.crt\"" Jul 23 15:46:27.820498 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.820489 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"default-dockercfg-z8gxn\"" Jul 23 15:46:27.820646 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.820489 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"rate-test-model-290c54-kserve-self-signed-certs\"" Jul 23 15:46:27.820646 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.820475 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-290c54\"/\"kube-root-ca.crt\"" Jul 23 15:46:27.826622 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.826602 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp"] Jul 23 15:46:27.864839 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.864816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-dshm\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.864954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.864846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-home\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.864954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.864870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-model-cache\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.864954 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.864900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tls-certs\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.865077 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.865017 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tmp-dir\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.865077 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.865055 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmkpk\" (UniqueName: \"kubernetes.io/projected/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-kube-api-access-hmkpk\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966286 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966256 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-dshm\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966286 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966287 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-home\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966510 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966309 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-model-cache\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966510 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966438 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tls-certs\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966510 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966509 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tmp-dir\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966680 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmkpk\" (UniqueName: \"kubernetes.io/projected/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-kube-api-access-hmkpk\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966680 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966659 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-model-cache\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966780 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966709 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-home\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.966877 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.966847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tmp-dir\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.968690 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.968669 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-dshm\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.968985 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.968966 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tls-certs\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:27.990376 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:27.990354 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmkpk\" (UniqueName: \"kubernetes.io/projected/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-kube-api-access-hmkpk\") pod \"rate-test-model-290c54-kserve-5b4b6884c-8dgkp\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:28.126661 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:28.126603 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:28.261562 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:28.261539 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp"] Jul 23 15:46:28.262883 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:46:28.262860 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod854e29ae_a3de_4c7c_8ba0_6cb6d9c5f513.slice/crio-ef2e8191cb9dccd384f2f7420a19b35120adf92fcf90d7309c49bab8f620afef WatchSource:0}: Error finding container ef2e8191cb9dccd384f2f7420a19b35120adf92fcf90d7309c49bab8f620afef: Status 404 returned error can't find the container with id ef2e8191cb9dccd384f2f7420a19b35120adf92fcf90d7309c49bab8f620afef Jul 23 15:46:28.338473 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:28.338437 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" event={"ID":"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513","Type":"ContainerStarted","Data":"cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171"} Jul 23 15:46:28.338578 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:28.338492 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" event={"ID":"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513","Type":"ContainerStarted","Data":"ef2e8191cb9dccd384f2f7420a19b35120adf92fcf90d7309c49bab8f620afef"} Jul 23 15:46:28.356997 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:28.356954 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" podStartSLOduration=1.356940788 podStartE2EDuration="1.356940788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:46:28.354936536 +0000 UTC m=+2775.948387456" watchObservedRunningTime="2026-07-23 15:46:28.356940788 +0000 UTC m=+2775.950391707" Jul 23 15:46:38.127697 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:38.127615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:38.127697 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:38.127664 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:38.139906 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:38.139881 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:38.404583 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:38.404497 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:46:53.488135 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.488101 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv"] Jul 23 15:46:53.491655 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.491633 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.494334 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.494314 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"openshift-service-ca.crt\"" Jul 23 15:46:53.495258 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.495241 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"kube-root-ca.crt\"" Jul 23 15:46:53.495392 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.495376 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"rate-test-model-d05e2c-kserve-self-signed-certs\"" Jul 23 15:46:53.495482 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.495466 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d05e2c\"/\"default-dockercfg-pkvfj\"" Jul 23 15:46:53.502162 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.502142 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv"] Jul 23 15:46:53.668743 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.668712 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tmp-dir\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.668743 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.668743 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-dshm\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.669053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.668762 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-model-cache\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.669053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.668783 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tls-certs\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.669053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.668843 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcpvq\" (UniqueName: \"kubernetes.io/projected/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-kube-api-access-tcpvq\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.669053 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.668944 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-home\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770119 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770052 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-home\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770226 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tmp-dir\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770226 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770177 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-dshm\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770226 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770207 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-model-cache\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770369 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tls-certs\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770369 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcpvq\" (UniqueName: \"kubernetes.io/projected/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-kube-api-access-tcpvq\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770495 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770477 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-home\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770550 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770490 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tmp-dir\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.770606 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.770543 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-model-cache\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.772398 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.772369 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-dshm\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.772819 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.772803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tls-certs\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.778900 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.778883 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcpvq\" (UniqueName: \"kubernetes.io/projected/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-kube-api-access-tcpvq\") pod \"rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.802088 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.802062 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:46:53.934891 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:53.934864 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv"] Jul 23 15:46:53.936509 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:46:53.936485 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a1d7055_2d21_4429_a93e_1fd7bd8f1e3a.slice/crio-c32e8819f86d08cc5cf15bd0e65d3f7843e7fd24181896ca59b89709823a030b WatchSource:0}: Error finding container c32e8819f86d08cc5cf15bd0e65d3f7843e7fd24181896ca59b89709823a030b: Status 404 returned error can't find the container with id c32e8819f86d08cc5cf15bd0e65d3f7843e7fd24181896ca59b89709823a030b Jul 23 15:46:54.480351 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:54.480321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" event={"ID":"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a","Type":"ContainerStarted","Data":"ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507"} Jul 23 15:46:54.480520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:54.480360 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" event={"ID":"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a","Type":"ContainerStarted","Data":"c32e8819f86d08cc5cf15bd0e65d3f7843e7fd24181896ca59b89709823a030b"} Jul 23 15:46:54.498273 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:46:54.498226 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" podStartSLOduration=1.4982113240000001 podStartE2EDuration="1.498211324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:46:54.49751754 +0000 UTC m=+2802.090968460" watchObservedRunningTime="2026-07-23 15:46:54.498211324 +0000 UTC m=+2802.091662243" Jul 23 15:47:03.802636 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:03.802599 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:47:03.803119 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:03.802662 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:47:03.814776 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:03.814751 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:47:04.551075 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:04.551048 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:47:50.391024 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:50.390956 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp"] Jul 23 15:47:51.282420 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:51.282388 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv"] Jul 23 15:47:54.950354 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:54.950323 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-140687783284528"] Jul 23 15:47:54.953733 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:54.953714 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:54.956401 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:54.956379 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rgjms\"" Jul 23 15:47:54.966607 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:54.966584 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-140687783284528"] Jul 23 15:47:55.057028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:55.056977 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjdh4\" (UniqueName: \"kubernetes.io/projected/d8def907-296e-4c5a-9318-7ee24e197c32-kube-api-access-fjdh4\") pod \"test-curl-7749-140687783284528\" (UID: \"d8def907-296e-4c5a-9318-7ee24e197c32\") " pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:55.158304 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:55.158280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjdh4\" (UniqueName: \"kubernetes.io/projected/d8def907-296e-4c5a-9318-7ee24e197c32-kube-api-access-fjdh4\") pod \"test-curl-7749-140687783284528\" (UID: \"d8def907-296e-4c5a-9318-7ee24e197c32\") " pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:55.168247 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:55.168226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjdh4\" (UniqueName: \"kubernetes.io/projected/d8def907-296e-4c5a-9318-7ee24e197c32-kube-api-access-fjdh4\") pod \"test-curl-7749-140687783284528\" (UID: \"d8def907-296e-4c5a-9318-7ee24e197c32\") " pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:55.264843 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:55.264777 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:55.398395 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:55.398364 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-140687783284528"] Jul 23 15:47:55.399566 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:47:55.399522 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8def907_296e_4c5a_9318_7ee24e197c32.slice/crio-dda776328a2620bf2de64e0f50cce6aa5fdcbd6893edba7e94600c892dcf0ca6 WatchSource:0}: Error finding container dda776328a2620bf2de64e0f50cce6aa5fdcbd6893edba7e94600c892dcf0ca6: Status 404 returned error can't find the container with id dda776328a2620bf2de64e0f50cce6aa5fdcbd6893edba7e94600c892dcf0ca6 Jul 23 15:47:55.400851 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:55.400836 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 15:47:55.822891 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:55.822859 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-140687783284528" event={"ID":"d8def907-296e-4c5a-9318-7ee24e197c32","Type":"ContainerStarted","Data":"dda776328a2620bf2de64e0f50cce6aa5fdcbd6893edba7e94600c892dcf0ca6"} Jul 23 15:47:56.829782 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:56.829742 2568 generic.go:356] "Generic (PLEG): container finished" podID="d8def907-296e-4c5a-9318-7ee24e197c32" containerID="603b86d894293ce401a6b3520859590cb81970a294b7efb44dc74b7a3c733760" exitCode=0 Jul 23 15:47:56.830197 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:56.829798 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-140687783284528" event={"ID":"d8def907-296e-4c5a-9318-7ee24e197c32","Type":"ContainerDied","Data":"603b86d894293ce401a6b3520859590cb81970a294b7efb44dc74b7a3c733760"} Jul 23 15:47:57.967884 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:57.967862 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:58.008339 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.008320 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7749-140687783284528_d8def907-296e-4c5a-9318-7ee24e197c32/test-curl-7749-140687783284528/0.log" Jul 23 15:47:58.034740 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.034717 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-140687783284528"] Jul 23 15:47:58.038418 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.038398 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-140687783284528"] Jul 23 15:47:58.085168 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.085119 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjdh4\" (UniqueName: \"kubernetes.io/projected/d8def907-296e-4c5a-9318-7ee24e197c32-kube-api-access-fjdh4\") pod \"d8def907-296e-4c5a-9318-7ee24e197c32\" (UID: \"d8def907-296e-4c5a-9318-7ee24e197c32\") " Jul 23 15:47:58.087250 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.087222 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8def907-296e-4c5a-9318-7ee24e197c32-kube-api-access-fjdh4" (OuterVolumeSpecName: "kube-api-access-fjdh4") pod "d8def907-296e-4c5a-9318-7ee24e197c32" (UID: "d8def907-296e-4c5a-9318-7ee24e197c32"). InnerVolumeSpecName "kube-api-access-fjdh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:47:58.185938 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.185913 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjdh4\" (UniqueName: \"kubernetes.io/projected/d8def907-296e-4c5a-9318-7ee24e197c32-kube-api-access-fjdh4\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:47:58.841605 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.841580 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dda776328a2620bf2de64e0f50cce6aa5fdcbd6893edba7e94600c892dcf0ca6" Jul 23 15:47:58.841772 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.841615 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:58.851676 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:47:58.851648 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-140687783284528\" is forbidden: User \"system:node:ip-10-0-139-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-2.ec2.internal' and this object" podUID="d8def907-296e-4c5a-9318-7ee24e197c32" pod="opendatahub/test-curl-7749-140687783284528" Jul 23 15:47:58.970525 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:47:58.970494 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8def907-296e-4c5a-9318-7ee24e197c32" path="/var/lib/kubelet/pods/d8def907-296e-4c5a-9318-7ee24e197c32/volumes" Jul 23 15:48:05.411159 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.411043 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" podUID="854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" containerName="main" containerID="cri-o://cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171" gracePeriod=45 Jul 23 15:48:05.669526 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.669467 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:48:05.747294 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747260 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-home\") pod \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " Jul 23 15:48:05.747440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747308 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-dshm\") pod \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " Jul 23 15:48:05.747440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747366 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tmp-dir\") pod \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " Jul 23 15:48:05.747440 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747409 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tls-certs\") pod \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " Jul 23 15:48:05.747608 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747453 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmkpk\" (UniqueName: \"kubernetes.io/projected/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-kube-api-access-hmkpk\") pod \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " Jul 23 15:48:05.747608 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747512 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-model-cache\") pod \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\" (UID: \"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513\") " Jul 23 15:48:05.747608 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747538 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-home" (OuterVolumeSpecName: "home") pod "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" (UID: "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:05.747754 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747711 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" (UID: "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:05.747876 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747851 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:05.747983 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747884 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:05.747983 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.747849 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-model-cache" (OuterVolumeSpecName: "model-cache") pod "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" (UID: "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:05.749545 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.749522 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" (UID: "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:48:05.749889 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.749872 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-kube-api-access-hmkpk" (OuterVolumeSpecName: "kube-api-access-hmkpk") pod "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" (UID: "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513"). InnerVolumeSpecName "kube-api-access-hmkpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:48:05.749889 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.749874 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-dshm" (OuterVolumeSpecName: "dshm") pod "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" (UID: "854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:05.848709 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.848690 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:05.848709 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.848711 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmkpk\" (UniqueName: \"kubernetes.io/projected/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-kube-api-access-hmkpk\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:05.848841 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.848721 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:05.848841 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.848731 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:05.888698 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.888671 2568 generic.go:356] "Generic (PLEG): container finished" podID="854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" containerID="cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171" exitCode=0 Jul 23 15:48:05.888809 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.888719 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" Jul 23 15:48:05.888809 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.888763 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" event={"ID":"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513","Type":"ContainerDied","Data":"cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171"} Jul 23 15:48:05.888948 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.888831 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp" event={"ID":"854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513","Type":"ContainerDied","Data":"ef2e8191cb9dccd384f2f7420a19b35120adf92fcf90d7309c49bab8f620afef"} Jul 23 15:48:05.888948 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.888857 2568 scope.go:117] "RemoveContainer" containerID="cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171" Jul 23 15:48:05.897892 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.897874 2568 scope.go:117] "RemoveContainer" containerID="cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171" Jul 23 15:48:05.898140 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:48:05.898121 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171\": container with ID starting with cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171 not found: ID does not exist" containerID="cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171" Jul 23 15:48:05.898193 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.898148 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171"} err="failed to get container status \"cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171\": rpc error: code = NotFound desc = could not find container \"cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171\": container with ID starting with cbb51ebf0f0bd6119c889bc28ad7ca81bb27a1ea46ca2c8fedf99eeaaee41171 not found: ID does not exist" Jul 23 15:48:05.911869 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.911846 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp"] Jul 23 15:48:05.917747 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:05.917728 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-290c54/rate-test-model-290c54-kserve-5b4b6884c-8dgkp"] Jul 23 15:48:06.302725 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.302681 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" podUID="9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" containerName="main" containerID="cri-o://ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507" gracePeriod=45 Jul 23 15:48:06.555289 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.555236 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:48:06.655842 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.655817 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-home\") pod \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " Jul 23 15:48:06.656020 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.655865 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tls-certs\") pod \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " Jul 23 15:48:06.656020 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.655899 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-model-cache\") pod \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " Jul 23 15:48:06.656020 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.655939 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tmp-dir\") pod \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " Jul 23 15:48:06.656020 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.655973 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-dshm\") pod \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " Jul 23 15:48:06.656264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.656089 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-home" (OuterVolumeSpecName: "home") pod "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" (UID: "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:06.656264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.656205 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" (UID: "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:06.656264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.656221 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" (UID: "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:06.656264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.656216 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcpvq\" (UniqueName: \"kubernetes.io/projected/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-kube-api-access-tcpvq\") pod \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\" (UID: \"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a\") " Jul 23 15:48:06.656534 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.656507 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tmp-dir\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:06.656534 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.656527 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-home\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:06.656648 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.656540 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-model-cache\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:06.658136 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.658107 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" (UID: "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 15:48:06.658264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.658163 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-dshm" (OuterVolumeSpecName: "dshm") pod "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" (UID: "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 15:48:06.658264 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.658195 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-kube-api-access-tcpvq" (OuterVolumeSpecName: "kube-api-access-tcpvq") pod "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" (UID: "9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a"). InnerVolumeSpecName "kube-api-access-tcpvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:48:06.757412 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.757389 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-tls-certs\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:06.757412 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.757409 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-dshm\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:06.757564 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.757418 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcpvq\" (UniqueName: \"kubernetes.io/projected/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a-kube-api-access-tcpvq\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:06.896114 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.896041 2568 generic.go:356] "Generic (PLEG): container finished" podID="9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" containerID="ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507" exitCode=0 Jul 23 15:48:06.896114 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.896080 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" Jul 23 15:48:06.896291 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.896130 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" event={"ID":"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a","Type":"ContainerDied","Data":"ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507"} Jul 23 15:48:06.896291 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.896168 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv" event={"ID":"9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a","Type":"ContainerDied","Data":"c32e8819f86d08cc5cf15bd0e65d3f7843e7fd24181896ca59b89709823a030b"} Jul 23 15:48:06.896291 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.896190 2568 scope.go:117] "RemoveContainer" containerID="ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507" Jul 23 15:48:06.905325 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.905309 2568 scope.go:117] "RemoveContainer" containerID="ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507" Jul 23 15:48:06.905577 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:48:06.905544 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507\": container with ID starting with ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507 not found: ID does not exist" containerID="ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507" Jul 23 15:48:06.905639 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.905585 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507"} err="failed to get container status \"ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507\": rpc error: code = NotFound desc = could not find container \"ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507\": container with ID starting with ccfaad6aff4d6b4c41109587b3cee84ff2ea60a5c47c0071a2da34d4064a8507 not found: ID does not exist" Jul 23 15:48:06.917420 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.917399 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv"] Jul 23 15:48:06.923372 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.923353 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d05e2c/rate-test-model-d05e2c-kserve-7f7cf9575f-qjthv"] Jul 23 15:48:06.963734 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.963712 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" path="/var/lib/kubelet/pods/854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513/volumes" Jul 23 15:48:06.964104 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:06.964091 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" path="/var/lib/kubelet/pods/9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a/volumes" Jul 23 15:48:07.000927 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.000907 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7749-140687783331856"] Jul 23 15:48:07.001300 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001287 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" containerName="main" Jul 23 15:48:07.001347 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001302 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" containerName="main" Jul 23 15:48:07.001347 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001316 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8def907-296e-4c5a-9318-7ee24e197c32" containerName="test-curl-7749-140687783284528" Jul 23 15:48:07.001347 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001323 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8def907-296e-4c5a-9318-7ee24e197c32" containerName="test-curl-7749-140687783284528" Jul 23 15:48:07.001347 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001340 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" containerName="main" Jul 23 15:48:07.001347 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001345 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" containerName="main" Jul 23 15:48:07.001497 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001400 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1d7055-2d21-4429-a93e-1fd7bd8f1e3a" containerName="main" Jul 23 15:48:07.001497 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001408 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8def907-296e-4c5a-9318-7ee24e197c32" containerName="test-curl-7749-140687783284528" Jul 23 15:48:07.001497 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.001415 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="854e29ae-a3de-4c7c-8ba0-6cb6d9c5f513" containerName="main" Jul 23 15:48:07.005562 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.005549 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:07.008262 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.008242 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rgjms\"" Jul 23 15:48:07.011964 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.011946 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-140687783331856"] Jul 23 15:48:07.060657 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.060627 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6bpv\" (UniqueName: \"kubernetes.io/projected/81f79f40-30f6-4463-8314-89523702d9e1-kube-api-access-t6bpv\") pod \"test-curl-7749-140687783331856\" (UID: \"81f79f40-30f6-4463-8314-89523702d9e1\") " pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:07.161306 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.161249 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6bpv\" (UniqueName: \"kubernetes.io/projected/81f79f40-30f6-4463-8314-89523702d9e1-kube-api-access-t6bpv\") pod \"test-curl-7749-140687783331856\" (UID: \"81f79f40-30f6-4463-8314-89523702d9e1\") " pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:07.170812 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.170792 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6bpv\" (UniqueName: \"kubernetes.io/projected/81f79f40-30f6-4463-8314-89523702d9e1-kube-api-access-t6bpv\") pod \"test-curl-7749-140687783331856\" (UID: \"81f79f40-30f6-4463-8314-89523702d9e1\") " pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:07.315631 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.315610 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:07.447655 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.447632 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7749-140687783331856"] Jul 23 15:48:07.449401 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:48:07.449360 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81f79f40_30f6_4463_8314_89523702d9e1.slice/crio-c884249bbb0754734105b9e623b60694142d084bb3dfc016ed8b5daaa910bfda WatchSource:0}: Error finding container c884249bbb0754734105b9e623b60694142d084bb3dfc016ed8b5daaa910bfda: Status 404 returned error can't find the container with id c884249bbb0754734105b9e623b60694142d084bb3dfc016ed8b5daaa910bfda Jul 23 15:48:07.903858 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.903831 2568 generic.go:356] "Generic (PLEG): container finished" podID="81f79f40-30f6-4463-8314-89523702d9e1" containerID="1e174a8ad096e1dc2fff4f8d2cabba3542375fd3aa126d39ae716e3c35c06681" exitCode=0 Jul 23 15:48:07.904174 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.903874 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-140687783331856" event={"ID":"81f79f40-30f6-4463-8314-89523702d9e1","Type":"ContainerDied","Data":"1e174a8ad096e1dc2fff4f8d2cabba3542375fd3aa126d39ae716e3c35c06681"} Jul 23 15:48:07.904174 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:07.903900 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7749-140687783331856" event={"ID":"81f79f40-30f6-4463-8314-89523702d9e1","Type":"ContainerStarted","Data":"c884249bbb0754734105b9e623b60694142d084bb3dfc016ed8b5daaa910bfda"} Jul 23 15:48:09.044470 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.044447 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:09.064107 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.064081 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7749-140687783331856_81f79f40-30f6-4463-8314-89523702d9e1/test-curl-7749-140687783331856/0.log" Jul 23 15:48:09.077415 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.077388 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6bpv\" (UniqueName: \"kubernetes.io/projected/81f79f40-30f6-4463-8314-89523702d9e1-kube-api-access-t6bpv\") pod \"81f79f40-30f6-4463-8314-89523702d9e1\" (UID: \"81f79f40-30f6-4463-8314-89523702d9e1\") " Jul 23 15:48:09.079310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.079289 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f79f40-30f6-4463-8314-89523702d9e1-kube-api-access-t6bpv" (OuterVolumeSpecName: "kube-api-access-t6bpv") pod "81f79f40-30f6-4463-8314-89523702d9e1" (UID: "81f79f40-30f6-4463-8314-89523702d9e1"). InnerVolumeSpecName "kube-api-access-t6bpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 15:48:09.087669 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.087631 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7749-140687783331856"] Jul 23 15:48:09.089849 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.089833 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7749-140687783331856"] Jul 23 15:48:09.178402 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.178345 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6bpv\" (UniqueName: \"kubernetes.io/projected/81f79f40-30f6-4463-8314-89523702d9e1-kube-api-access-t6bpv\") on node \"ip-10-0-139-2.ec2.internal\" DevicePath \"\"" Jul 23 15:48:09.917943 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.917913 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:09.918139 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:09.917959 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c884249bbb0754734105b9e623b60694142d084bb3dfc016ed8b5daaa910bfda" Jul 23 15:48:09.927934 ip-10-0-139-2 kubenswrapper[2568]: E0723 15:48:09.927909 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7749-140687783331856\" is forbidden: User \"system:node:ip-10-0-139-2.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-2.ec2.internal' and this object" podUID="81f79f40-30f6-4463-8314-89523702d9e1" pod="opendatahub/test-curl-7749-140687783331856" Jul 23 15:48:10.963887 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:48:10.963857 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f79f40-30f6-4463-8314-89523702d9e1" path="/var/lib/kubelet/pods/81f79f40-30f6-4463-8314-89523702d9e1/volumes" Jul 23 15:49:00.380659 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:00.380623 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dffjv_33a8aab2-13bd-4f1e-b053-4287cfdd5ad9/server/0.log" Jul 23 15:49:00.506377 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:00.506348 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gkw95_637001c1-bdf5-44ca-ad5e-4404f6af2c9d/manager/0.log" Jul 23 15:49:00.883566 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:00.883541 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75984d86c4-l4xn6_bb4b2618-c991-417e-bbec-8e4095c6ba54/copy-manifests/0.log" Jul 23 15:49:00.889968 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:00.889937 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75984d86c4-l4xn6_bb4b2618-c991-417e-bbec-8e4095c6ba54/manager/0.log" Jul 23 15:49:02.612813 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.612780 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n_05cf2b9c-944b-4e6a-8a08-315e1a603359/util/0.log" Jul 23 15:49:02.618889 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.618864 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n_05cf2b9c-944b-4e6a-8a08-315e1a603359/pull/0.log" Jul 23 15:49:02.624561 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.624522 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n_05cf2b9c-944b-4e6a-8a08-315e1a603359/extract/0.log" Jul 23 15:49:02.742050 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.742028 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95_37cb3764-8f3c-4b6c-8ed8-2734c0c08061/util/0.log" Jul 23 15:49:02.747973 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.747939 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95_37cb3764-8f3c-4b6c-8ed8-2734c0c08061/pull/0.log" Jul 23 15:49:02.754028 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.753984 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95_37cb3764-8f3c-4b6c-8ed8-2734c0c08061/extract/0.log" Jul 23 15:49:02.857903 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.857884 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5_a2549618-c352-44a2-8e00-5b50a5f38df3/util/0.log" Jul 23 15:49:02.864177 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.864126 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5_a2549618-c352-44a2-8e00-5b50a5f38df3/pull/0.log" Jul 23 15:49:02.869977 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.869962 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5_a2549618-c352-44a2-8e00-5b50a5f38df3/extract/0.log" Jul 23 15:49:02.978159 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.978136 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx_7a09ffcc-27ad-4e5e-945c-33c19a4cb43c/extract/0.log" Jul 23 15:49:02.983937 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.983919 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx_7a09ffcc-27ad-4e5e-945c-33c19a4cb43c/util/0.log" Jul 23 15:49:02.990256 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:02.990237 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx_7a09ffcc-27ad-4e5e-945c-33c19a4cb43c/pull/0.log" Jul 23 15:49:03.475766 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:03.475739 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dncdp_801a2923-1a42-4ba8-8876-4bdf512cfba5/kuadrant-console-plugin/0.log" Jul 23 15:49:03.740906 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:03.740825 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-4cq4c_a66e7b80-8997-47e5-85bd-3f9b6061f902/manager/0.log" Jul 23 15:49:04.600953 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:04.600909 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848d77456-5lchx_c383a82b-38ec-42dd-9be0-de48c82d87e0/kube-auth-proxy/0.log" Jul 23 15:49:05.917511 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:05.917476 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fvhzm9_dbc5ec13-2bae-450b-85fe-d453f1fca63b/main/0.log" Jul 23 15:49:06.048577 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:06.048550 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-86q97_bb597cf9-51f2-4cea-bfb3-118330aa0aa5/main/0.log" Jul 23 15:49:06.196395 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:06.196365 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-j4d8p_610ddf56-6137-4634-af18-960d86b0eed2/main/0.log" Jul 23 15:49:12.039796 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.039748 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9l4h4/must-gather-w2ht2"] Jul 23 15:49:12.040214 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.040197 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f79f40-30f6-4463-8314-89523702d9e1" containerName="test-curl-7749-140687783331856" Jul 23 15:49:12.040214 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.040211 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f79f40-30f6-4463-8314-89523702d9e1" containerName="test-curl-7749-140687783331856" Jul 23 15:49:12.040300 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.040269 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f79f40-30f6-4463-8314-89523702d9e1" containerName="test-curl-7749-140687783331856" Jul 23 15:49:12.043472 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.043452 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.046110 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.046086 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9l4h4\"/\"kube-root-ca.crt\"" Jul 23 15:49:12.047005 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.046967 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9l4h4\"/\"default-dockercfg-bddm9\"" Jul 23 15:49:12.047133 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.046972 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9l4h4\"/\"openshift-service-ca.crt\"" Jul 23 15:49:12.051048 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.050820 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9l4h4/must-gather-w2ht2"] Jul 23 15:49:12.070812 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.070786 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fe410c0-3c88-48c6-99d2-7d9e242ac178-must-gather-output\") pod \"must-gather-w2ht2\" (UID: \"2fe410c0-3c88-48c6-99d2-7d9e242ac178\") " pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.070902 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.070829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9dvm\" (UniqueName: \"kubernetes.io/projected/2fe410c0-3c88-48c6-99d2-7d9e242ac178-kube-api-access-x9dvm\") pod \"must-gather-w2ht2\" (UID: \"2fe410c0-3c88-48c6-99d2-7d9e242ac178\") " pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.172124 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.172091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fe410c0-3c88-48c6-99d2-7d9e242ac178-must-gather-output\") pod \"must-gather-w2ht2\" (UID: \"2fe410c0-3c88-48c6-99d2-7d9e242ac178\") " pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.172248 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.172131 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9dvm\" (UniqueName: \"kubernetes.io/projected/2fe410c0-3c88-48c6-99d2-7d9e242ac178-kube-api-access-x9dvm\") pod \"must-gather-w2ht2\" (UID: \"2fe410c0-3c88-48c6-99d2-7d9e242ac178\") " pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.172445 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.172423 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2fe410c0-3c88-48c6-99d2-7d9e242ac178-must-gather-output\") pod \"must-gather-w2ht2\" (UID: \"2fe410c0-3c88-48c6-99d2-7d9e242ac178\") " pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.180375 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.180354 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9dvm\" (UniqueName: \"kubernetes.io/projected/2fe410c0-3c88-48c6-99d2-7d9e242ac178-kube-api-access-x9dvm\") pod \"must-gather-w2ht2\" (UID: \"2fe410c0-3c88-48c6-99d2-7d9e242ac178\") " pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.354342 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.354286 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9l4h4/must-gather-w2ht2" Jul 23 15:49:12.485491 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:12.485453 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9l4h4/must-gather-w2ht2"] Jul 23 15:49:12.487064 ip-10-0-139-2 kubenswrapper[2568]: W0723 15:49:12.487029 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fe410c0_3c88_48c6_99d2_7d9e242ac178.slice/crio-f86112bee48ba2b43e3b322c79809a424074832b124c535d0be59b9207fc77b2 WatchSource:0}: Error finding container f86112bee48ba2b43e3b322c79809a424074832b124c535d0be59b9207fc77b2: Status 404 returned error can't find the container with id f86112bee48ba2b43e3b322c79809a424074832b124c535d0be59b9207fc77b2 Jul 23 15:49:13.252160 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:13.252126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9l4h4/must-gather-w2ht2" event={"ID":"2fe410c0-3c88-48c6-99d2-7d9e242ac178","Type":"ContainerStarted","Data":"f86112bee48ba2b43e3b322c79809a424074832b124c535d0be59b9207fc77b2"} Jul 23 15:49:15.270405 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:15.270366 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9l4h4/must-gather-w2ht2" event={"ID":"2fe410c0-3c88-48c6-99d2-7d9e242ac178","Type":"ContainerStarted","Data":"4aab6f12df32efcbfdbbf559f7a50deec8f6405dcebc956c416fda0600b5a36d"} Jul 23 15:49:15.270815 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:15.270413 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9l4h4/must-gather-w2ht2" event={"ID":"2fe410c0-3c88-48c6-99d2-7d9e242ac178","Type":"ContainerStarted","Data":"ecd930fb74a38f692b586519a102c99b6ce2ae35732f3088ebc444b0d5af106f"} Jul 23 15:49:15.290551 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:15.290456 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9l4h4/must-gather-w2ht2" podStartSLOduration=1.293196775 podStartE2EDuration="3.290428413s" totalImagesPullingTime="1.997231638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 15:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:15.286672241 +0000 UTC m=+2942.880123161" watchObservedRunningTime="2026-07-23 15:49:15.290428413 +0000 UTC m=+2942.883879333" Jul 23 15:49:16.127535 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:16.127506 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g7spv_00c24181-cecc-4735-aa71-d217f7dcc3b1/global-pull-secret-syncer/0.log" Jul 23 15:49:16.282639 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:16.282605 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mprlv_a93bd9a2-2c0c-43d1-b80a-afcb17d9a49e/konnectivity-agent/0.log" Jul 23 15:49:16.328445 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:16.328414 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-2.ec2.internal_eeaa75121a38762b69786f5043319e69/haproxy/0.log" Jul 23 15:49:20.408978 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.408943 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n_05cf2b9c-944b-4e6a-8a08-315e1a603359/extract/0.log" Jul 23 15:49:20.425198 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.425170 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n_05cf2b9c-944b-4e6a-8a08-315e1a603359/util/0.log" Jul 23 15:49:20.447823 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.447795 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ktk8n_05cf2b9c-944b-4e6a-8a08-315e1a603359/pull/0.log" Jul 23 15:49:20.473520 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.473487 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95_37cb3764-8f3c-4b6c-8ed8-2734c0c08061/extract/0.log" Jul 23 15:49:20.495902 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.495872 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95_37cb3764-8f3c-4b6c-8ed8-2734c0c08061/util/0.log" Jul 23 15:49:20.518559 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.518378 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0j7m95_37cb3764-8f3c-4b6c-8ed8-2734c0c08061/pull/0.log" Jul 23 15:49:20.558913 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.558889 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5_a2549618-c352-44a2-8e00-5b50a5f38df3/extract/0.log" Jul 23 15:49:20.583341 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.583318 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5_a2549618-c352-44a2-8e00-5b50a5f38df3/util/0.log" Jul 23 15:49:20.608037 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.608013 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l85d5_a2549618-c352-44a2-8e00-5b50a5f38df3/pull/0.log" Jul 23 15:49:20.635711 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.635680 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx_7a09ffcc-27ad-4e5e-945c-33c19a4cb43c/extract/0.log" Jul 23 15:49:20.658098 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.658068 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx_7a09ffcc-27ad-4e5e-945c-33c19a4cb43c/util/0.log" Jul 23 15:49:20.680510 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:20.680441 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sjnhx_7a09ffcc-27ad-4e5e-945c-33c19a4cb43c/pull/0.log" Jul 23 15:49:21.121355 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:21.121328 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dncdp_801a2923-1a42-4ba8-8876-4bdf512cfba5/kuadrant-console-plugin/0.log" Jul 23 15:49:21.381381 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:21.380634 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-4cq4c_a66e7b80-8997-47e5-85bd-3f9b6061f902/manager/0.log" Jul 23 15:49:23.485797 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:23.485767 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jqrct_ea18f270-2af0-4da5-aba8-bdb323a89d6f/kube-state-metrics/0.log" Jul 23 15:49:23.524340 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:23.524313 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jqrct_ea18f270-2af0-4da5-aba8-bdb323a89d6f/kube-rbac-proxy-main/0.log" Jul 23 15:49:23.550801 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:23.550774 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-jqrct_ea18f270-2af0-4da5-aba8-bdb323a89d6f/kube-rbac-proxy-self/0.log" Jul 23 15:49:23.588586 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:23.587940 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-876bc649b-f8bq8_3b9f08ca-1022-45fd-8943-e549ba656172/metrics-server/0.log" Jul 23 15:49:23.745090 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:23.745016 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wq85_664f3dfa-35fe-4428-99ee-da94508b1d05/node-exporter/0.log" Jul 23 15:49:23.767217 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:23.767170 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wq85_664f3dfa-35fe-4428-99ee-da94508b1d05/kube-rbac-proxy/0.log" Jul 23 15:49:23.793009 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:23.792959 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9wq85_664f3dfa-35fe-4428-99ee-da94508b1d05/init-textfile/0.log" Jul 23 15:49:24.249005 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:24.248965 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-pcbm2_3d4deab9-b559-49d6-a1eb-d21d6f9111bd/prometheus-operator-admission-webhook/0.log" Jul 23 15:49:24.371213 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:24.371180 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766b9c6d48-sjchg_fc849a00-f7c0-49b5-81aa-1fda9417c5ba/thanos-query/0.log" Jul 23 15:49:24.395694 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:24.395630 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766b9c6d48-sjchg_fc849a00-f7c0-49b5-81aa-1fda9417c5ba/kube-rbac-proxy-web/0.log" Jul 23 15:49:24.419378 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:24.419348 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766b9c6d48-sjchg_fc849a00-f7c0-49b5-81aa-1fda9417c5ba/kube-rbac-proxy/0.log" Jul 23 15:49:24.445164 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:24.445131 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766b9c6d48-sjchg_fc849a00-f7c0-49b5-81aa-1fda9417c5ba/prom-label-proxy/0.log" Jul 23 15:49:24.466902 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:24.466881 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766b9c6d48-sjchg_fc849a00-f7c0-49b5-81aa-1fda9417c5ba/kube-rbac-proxy-rules/0.log" Jul 23 15:49:24.485885 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:24.485826 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766b9c6d48-sjchg_fc849a00-f7c0-49b5-81aa-1fda9417c5ba/kube-rbac-proxy-metrics/0.log" Jul 23 15:49:25.662048 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:25.662020 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-87q9q_389cac9f-022f-4cce-a874-ef6a70b150ac/networking-console-plugin/0.log" Jul 23 15:49:26.268435 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.268408 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/1.log" Jul 23 15:49:26.277081 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.277049 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j56c4_08654902-ac10-401e-b4a7-f3c6e5055acc/console-operator/2.log" Jul 23 15:49:26.335891 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.335855 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7"] Jul 23 15:49:26.344384 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.344356 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.349714 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.349690 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7"] Jul 23 15:49:26.418779 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.418737 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-podres\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.418967 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.418819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9kd\" (UniqueName: \"kubernetes.io/projected/90bb316f-4420-4680-b6a3-54ec11c202af-kube-api-access-mv9kd\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.418967 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.418858 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-proc\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.418967 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.418886 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-sys\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.418967 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.418937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-lib-modules\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520305 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520223 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-lib-modules\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520305 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-podres\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520324 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9kd\" (UniqueName: \"kubernetes.io/projected/90bb316f-4420-4680-b6a3-54ec11c202af-kube-api-access-mv9kd\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520343 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-proc\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-sys\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520444 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-lib-modules\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-podres\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520533 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520523 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-sys\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.520791 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.520557 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90bb316f-4420-4680-b6a3-54ec11c202af-proc\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.529439 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.529408 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9kd\" (UniqueName: \"kubernetes.io/projected/90bb316f-4420-4680-b6a3-54ec11c202af-kube-api-access-mv9kd\") pod \"perf-node-gather-daemonset-j7vl7\" (UID: \"90bb316f-4420-4680-b6a3-54ec11c202af\") " pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.663480 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.663450 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:26.857514 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.856426 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7"] Jul 23 15:49:26.932555 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:26.932503 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-dgg29_c42d3923-b5f6-48c5-913d-1aec82494188/download-server/0.log" Jul 23 15:49:27.373941 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:27.373886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" event={"ID":"90bb316f-4420-4680-b6a3-54ec11c202af","Type":"ContainerStarted","Data":"6618f6588f6e5bad25b74b7b66a365976ff8379b17dd97d9c07e39e21bf1f797"} Jul 23 15:49:27.374160 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:27.373959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" event={"ID":"90bb316f-4420-4680-b6a3-54ec11c202af","Type":"ContainerStarted","Data":"4e2f47e916adf88c26f9c683125dfcd89631ff3352b46850fa12d7f1b5c0d275"} Jul 23 15:49:27.374386 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:27.374363 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:27.394283 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:27.394221 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" podStartSLOduration=1.3942002279999999 podStartE2EDuration="1.394200228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 15:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 15:49:27.390652641 +0000 UTC m=+2954.984103562" watchObservedRunningTime="2026-07-23 15:49:27.394200228 +0000 UTC m=+2954.987651148" Jul 23 15:49:28.624310 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:28.624275 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tq87x_0a3024aa-1fca-4b32-825a-ac2b80fa4251/dns/0.log" Jul 23 15:49:28.644478 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:28.644429 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tq87x_0a3024aa-1fca-4b32-825a-ac2b80fa4251/kube-rbac-proxy/0.log" Jul 23 15:49:28.697718 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:28.697675 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2tnf9_0b36d684-b669-45b1-b436-4b2d52a6134a/dns-node-resolver/0.log" Jul 23 15:49:29.433468 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:29.433439 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vwksz_66136ccc-448f-4364-83e7-df4ba4703e61/node-ca/0.log" Jul 23 15:49:30.615770 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:30.615741 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848d77456-5lchx_c383a82b-38ec-42dd-9be0-de48c82d87e0/kube-auth-proxy/0.log" Jul 23 15:49:31.544730 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:31.544694 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8792c_1af26b3d-e245-494b-b21b-5cb771291790/serve-healthcheck-canary/0.log" Jul 23 15:49:32.027083 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:32.027050 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-lf7g5_a45bae56-0ce9-4902-ba2a-604c879907cd/insights-operator/0.log" Jul 23 15:49:32.028024 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:32.027969 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-lf7g5_a45bae56-0ce9-4902-ba2a-604c879907cd/insights-operator/1.log" Jul 23 15:49:32.189744 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:32.189715 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lvr6j_fcd2b199-7aac-46f3-a9fc-e7514d889860/kube-rbac-proxy/0.log" Jul 23 15:49:32.209808 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:32.209784 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lvr6j_fcd2b199-7aac-46f3-a9fc-e7514d889860/exporter/0.log" Jul 23 15:49:32.230468 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:32.230445 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lvr6j_fcd2b199-7aac-46f3-a9fc-e7514d889860/extractor/0.log" Jul 23 15:49:33.394476 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:33.394449 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9l4h4/perf-node-gather-daemonset-j7vl7" Jul 23 15:49:34.878015 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:34.877946 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dffjv_33a8aab2-13bd-4f1e-b053-4287cfdd5ad9/server/0.log" Jul 23 15:49:34.995956 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:34.995881 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gkw95_637001c1-bdf5-44ca-ad5e-4404f6af2c9d/manager/0.log" Jul 23 15:49:35.182047 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:35.182016 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75984d86c4-l4xn6_bb4b2618-c991-417e-bbec-8e4095c6ba54/manager/0.log" Jul 23 15:49:35.200153 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:35.200118 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-75984d86c4-l4xn6_bb4b2618-c991-417e-bbec-8e4095c6ba54/copy-manifests/0.log" Jul 23 15:49:36.597194 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:36.597152 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68b858458d-nm2vp_d4882f36-f4e0-4f47-a9f4-00b9822fded6/manager/0.log" Jul 23 15:49:36.648597 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:36.648526 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848d77456-5lchx_c383a82b-38ec-42dd-9be0-de48c82d87e0/kube-auth-proxy/0.log" Jul 23 15:49:38.206089 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:38.206056 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-848d77456-5lchx_c383a82b-38ec-42dd-9be0-de48c82d87e0/kube-auth-proxy/0.log" Jul 23 15:49:41.738918 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:41.738888 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-69l6l_7924e63c-f6ec-4008-bc30-2d35fc835b7c/kube-storage-version-migrator-operator/1.log" Jul 23 15:49:41.740879 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:41.740854 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-69l6l_7924e63c-f6ec-4008-bc30-2d35fc835b7c/kube-storage-version-migrator-operator/0.log" Jul 23 15:49:42.925707 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:42.925672 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95l8l_2df20e09-01a7-488a-aaa8-1ce22e207283/kube-multus-additional-cni-plugins/0.log" Jul 23 15:49:42.945578 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:42.945552 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95l8l_2df20e09-01a7-488a-aaa8-1ce22e207283/egress-router-binary-copy/0.log" Jul 23 15:49:42.965801 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:42.965774 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95l8l_2df20e09-01a7-488a-aaa8-1ce22e207283/cni-plugins/0.log" Jul 23 15:49:42.986711 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:42.986688 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95l8l_2df20e09-01a7-488a-aaa8-1ce22e207283/bond-cni-plugin/0.log" Jul 23 15:49:43.005552 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:43.005536 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95l8l_2df20e09-01a7-488a-aaa8-1ce22e207283/routeoverride-cni/0.log" Jul 23 15:49:43.026884 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:43.026859 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95l8l_2df20e09-01a7-488a-aaa8-1ce22e207283/whereabouts-cni-bincopy/0.log" Jul 23 15:49:43.044692 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:43.044672 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-95l8l_2df20e09-01a7-488a-aaa8-1ce22e207283/whereabouts-cni/0.log" Jul 23 15:49:43.257942 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:43.257911 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rsrjp_f66b5f40-43fa-4d2e-a84f-da5a18d81b52/kube-multus/0.log" Jul 23 15:49:43.322231 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:43.322199 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f6mzs_954165e8-54ba-4595-b41a-186d83bd2818/network-metrics-daemon/0.log" Jul 23 15:49:43.340735 ip-10-0-139-2 kubenswrapper[2568]: I0723 15:49:43.340704 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f6mzs_954165e8-54ba-4595-b41a-186d83bd2818/kube-rbac-proxy/0.log"