Jul 07 18:59:02.858279 ip-10-0-137-8 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 07 18:59:02.858288 ip-10-0-137-8 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 07 18:59:02.858295 ip-10-0-137-8 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 07 18:59:02.858628 ip-10-0-137-8 systemd[1]: Failed to start Kubernetes Kubelet. Jul 07 18:59:13.011967 ip-10-0-137-8 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 07 18:59:13.011986 ip-10-0-137-8 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 25a4ce37e1904605b3cf010d393da396 -- Jul 07 19:01:14.841531 ip-10-0-137-8 systemd[1]: Starting Kubernetes Kubelet... Jul 07 19:01:15.315876 ip-10-0-137-8 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:01:15.315876 ip-10-0-137-8 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 19:01:15.315876 ip-10-0-137-8 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:01:15.315876 ip-10-0-137-8 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 19:01:15.315876 ip-10-0-137-8 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 19:01:15.316823 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.316723 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 19:01:15.321509 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321484 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:01:15.321509 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321507 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:01:15.321509 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321511 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321517 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321522 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321525 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321529 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321532 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321535 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321538 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321542 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321545 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321547 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321550 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321553 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321557 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321559 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321562 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321569 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321573 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321575 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:01:15.321616 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321578 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321581 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321583 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321586 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321589 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321592 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321595 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321597 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321600 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321602 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321605 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321608 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321610 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321613 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321616 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321620 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321623 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321626 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321629 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321632 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:01:15.322093 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321635 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321637 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321640 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321642 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321645 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321647 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321651 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321653 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321656 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321659 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321661 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321664 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321667 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321678 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321682 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321685 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321688 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321691 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321694 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321697 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:01:15.322663 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321700 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321703 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321705 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321708 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321711 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321714 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321717 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321720 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321724 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321727 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321731 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321734 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321737 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321741 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321746 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321749 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321752 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321755 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321757 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:01:15.323183 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321761 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321764 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321766 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321769 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321772 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321774 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321777 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321779 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321783 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321786 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321789 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321791 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321794 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.321797 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.322642 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.322648 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.322652 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.322655 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.322660 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323469 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:01:15.323690 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323477 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323481 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323485 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323488 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323491 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323494 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323498 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323500 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323503 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323506 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323508 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323511 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323513 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323518 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323522 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323525 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323528 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323531 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323534 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323537 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:01:15.324192 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323540 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323543 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323545 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323548 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323551 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323553 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323557 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323559 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323562 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323564 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323567 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323570 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323573 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323577 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323580 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323583 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323586 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323589 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323591 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323594 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:01:15.324732 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323598 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323601 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323603 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323606 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323609 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323611 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323615 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323619 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323622 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323625 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323627 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323630 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323634 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323636 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323639 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323641 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323644 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323647 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323649 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:01:15.325271 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323652 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323655 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323659 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323662 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323665 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323668 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323671 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323673 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323676 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323681 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323685 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323688 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323691 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323694 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323697 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323700 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323703 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323706 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323709 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:01:15.325759 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323712 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323715 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323719 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323721 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323724 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323727 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323730 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323732 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323735 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323738 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323740 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323743 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323746 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323748 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323751 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323754 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323757 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323760 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323763 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.323765 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:01:15.326296 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324209 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324217 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324220 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324224 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324227 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324230 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324233 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324236 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324257 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324261 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324263 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324266 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324269 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324273 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324277 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324280 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324283 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324286 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324289 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324291 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:01:15.326837 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324294 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324297 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324299 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324302 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324305 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324308 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324310 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324313 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324316 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324318 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324323 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324327 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324330 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324333 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324336 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324339 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324341 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324344 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324347 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:01:15.327358 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324349 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324352 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324355 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324358 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324361 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324367 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324369 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324377 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324380 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324383 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324385 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324388 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324391 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324394 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324396 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324399 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324402 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324404 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324407 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324410 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:01:15.327834 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324412 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324414 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324417 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324421 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324425 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324427 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324430 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324433 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324436 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324439 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324442 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324444 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324448 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324451 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324453 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324456 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324459 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324463 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324466 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:01:15.328385 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324469 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324473 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324475 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324478 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324481 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324483 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324486 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324489 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324492 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324494 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324497 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324501 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324504 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324506 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324509 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324512 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324514 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324517 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324520 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324523 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:01:15.328875 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.324525 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324604 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324612 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324618 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324623 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324627 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324631 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324635 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324640 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324644 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324649 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324653 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324656 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324660 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324664 2559 flags.go:64] FLAG: --cgroup-root="" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324667 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324670 2559 flags.go:64] FLAG: --client-ca-file="" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324674 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324677 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324686 2559 flags.go:64] FLAG: --cluster-domain="" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324689 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324692 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324696 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324699 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 19:01:15.329387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324703 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324707 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324710 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324713 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324717 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324720 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324723 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324726 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324729 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324734 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324737 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324740 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324744 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324748 2559 flags.go:64] FLAG: --enable-server="true" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324751 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324760 2559 flags.go:64] FLAG: --event-burst="100" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324763 2559 flags.go:64] FLAG: --event-qps="50" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324769 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324773 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324777 2559 flags.go:64] FLAG: --eviction-hard="" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324781 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324784 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324787 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324793 2559 flags.go:64] FLAG: --eviction-soft="" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324797 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 19:01:15.330028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324800 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324803 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324806 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324809 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324812 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324815 2559 flags.go:64] FLAG: --feature-gates="" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324819 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324823 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324826 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324829 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324833 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324836 2559 flags.go:64] FLAG: --help="false" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324839 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324842 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324845 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324848 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324852 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324855 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324858 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324861 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324864 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324867 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324870 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324873 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 19:01:15.330726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324877 2559 flags.go:64] FLAG: --kube-reserved="" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324882 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324885 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324889 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324892 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324895 2559 flags.go:64] FLAG: --lock-file="" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324898 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324902 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324905 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324911 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324914 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324917 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324920 2559 flags.go:64] FLAG: --logging-format="text" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324923 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324927 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324930 2559 flags.go:64] FLAG: --manifest-url="" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324933 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324938 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324941 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324945 2559 flags.go:64] FLAG: --max-pods="110" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324948 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324951 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324955 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324958 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324961 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 19:01:15.331373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324964 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324967 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324975 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324978 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324981 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324985 2559 flags.go:64] FLAG: --pod-cidr="" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324988 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.324994 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325000 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325003 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325006 2559 flags.go:64] FLAG: --port="10250" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325010 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325013 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02f31fe252ea04e32" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325016 2559 flags.go:64] FLAG: --qos-reserved="" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325020 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325023 2559 flags.go:64] FLAG: --register-node="true" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325026 2559 flags.go:64] FLAG: --register-with-taints="" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325035 2559 flags.go:64] FLAG: --registry-burst="10" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325038 2559 flags.go:64] FLAG: --registry-qps="5" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325041 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325044 2559 flags.go:64] FLAG: --reserved-memory="" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325048 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325051 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325054 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325057 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 19:01:15.332033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325060 2559 flags.go:64] FLAG: --runonce="false" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325063 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325066 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325070 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325073 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325076 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325079 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325083 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325086 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325089 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325092 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325095 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325098 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325101 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325104 2559 flags.go:64] FLAG: --system-cgroups="" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325114 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325119 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325122 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325125 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325132 2559 flags.go:64] FLAG: --tls-min-version="" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325135 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325138 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325141 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325147 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325151 2559 flags.go:64] FLAG: --v="2" Jul 07 19:01:15.332844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325155 2559 flags.go:64] FLAG: --version="false" Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325159 2559 flags.go:64] FLAG: --vmodule="" Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325164 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.325167 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325307 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325312 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325315 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325318 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325321 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325324 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325327 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325329 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325332 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325335 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325338 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325340 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325343 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325346 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325349 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325352 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:01:15.333515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325354 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325357 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325362 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325365 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325367 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325370 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325373 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325375 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325378 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325382 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325386 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325390 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325394 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325397 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325399 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325402 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325405 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325408 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325411 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325414 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:01:15.334065 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325416 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325419 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325422 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325424 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325427 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325430 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325432 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325435 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325438 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325440 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325443 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325446 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325449 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325453 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325455 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325458 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325461 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325464 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325466 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325469 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:01:15.334591 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325472 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325475 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325477 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325480 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325485 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325487 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325490 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325493 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325495 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325498 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325502 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325506 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325509 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325513 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325517 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325521 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325524 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325526 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325529 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:01:15.335110 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325532 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325535 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325537 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325540 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325543 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325545 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325548 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325551 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325554 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325556 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325559 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325561 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325564 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325567 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325569 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325572 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325575 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325577 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325581 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325583 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:01:15.335615 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325586 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325589 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325591 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.325594 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.326230 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.333827 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.333848 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.333868 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.333874 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333955 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333962 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333966 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333970 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333973 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333976 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:01:15.336133 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333979 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333982 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333985 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333988 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333990 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333993 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333996 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.333998 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334001 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334004 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334007 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334010 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334012 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334015 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334018 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334029 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334032 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334035 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334038 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334040 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:01:15.336549 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334043 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334046 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334048 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334052 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334055 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334057 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334060 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334063 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334066 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334068 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334071 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334074 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334079 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334083 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334087 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334090 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334093 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334096 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334099 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:01:15.337058 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334102 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334105 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334108 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334111 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334114 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334117 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334120 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334123 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334126 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334134 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334137 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334140 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334142 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334145 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334148 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334151 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334155 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334157 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334160 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334163 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:01:15.337706 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334166 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334169 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334171 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334174 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334177 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334180 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334183 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334185 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334188 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334191 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334193 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334197 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334201 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334204 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334207 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334214 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334217 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334219 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334222 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:01:15.338219 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334224 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334229 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334233 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334255 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334259 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334262 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334265 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334268 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334271 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334274 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334277 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334280 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334283 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334286 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:01:15.338724 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334289 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.334300 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334452 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334457 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334460 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334463 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334467 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334470 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334474 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334477 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334480 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334483 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334486 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334488 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 19:01:15.339211 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334491 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334494 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334497 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334500 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334502 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334505 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334508 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334511 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334519 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334522 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334525 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334527 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334530 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334533 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334535 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334538 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334541 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334544 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334546 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 19:01:15.339610 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334549 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334552 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334554 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334557 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334559 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334562 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334565 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334567 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334570 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334572 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334575 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334578 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334581 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334583 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334586 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334589 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334591 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334594 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334598 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 19:01:15.340347 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334601 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334604 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334606 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334615 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334618 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334621 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334624 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334627 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334630 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334633 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334635 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334639 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334643 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334646 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334649 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334652 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334654 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334659 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334662 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 19:01:15.341090 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334665 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334674 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334679 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334683 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334686 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334689 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334692 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334695 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334698 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334701 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334704 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334707 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334710 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334712 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334715 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334718 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334721 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334732 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334735 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334737 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 19:01:15.341700 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334740 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334743 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334746 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334749 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334751 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334754 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334757 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334760 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334763 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:15.334765 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.334770 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.334930 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.337114 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.338708 2559 server.go:1019] "Starting client certificate rotation" Jul 07 19:01:15.342452 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.338844 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:01:15.342938 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.338904 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 19:01:15.342938 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.342136 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:01:15.344151 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.344111 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 19:01:15.355472 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.355443 2559 log.go:25] "Validated CRI v1 runtime API" Jul 07 19:01:15.358710 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.358644 2559 log.go:25] "Validated CRI v1 image API" Jul 07 19:01:15.360391 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.360370 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 19:01:15.363032 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.363010 2559 fs.go:135] Filesystem UUIDs: map[4791bf98-f795-4538-9f0c-6209bcbcf4d2:/dev/nvme0n1p4 4e96691d-7e05-4eb0-b1c0-7b62d3656227:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 07 19:01:15.363093 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.363031 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 19:01:15.368857 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.368836 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:01:15.369448 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.369285 2559 manager.go:216] Machine: {Timestamp:2026-07-07 19:01:15.366979612 +0000 UTC m=+0.388629610 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200429 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26dde8c0f0d6165106f8c8551bfe8b SystemUUID:ec26dde8-c0f0-d616-5106-f8c8551bfe8b BootID:25a4ce37-e190-4605-b3cf-010d393da396 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9d:2b:a5:6c:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9d:2b:a5:6c:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:82:17:56:54:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 19:01:15.369448 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.369418 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 19:01:15.369625 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.369530 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 19:01:15.371995 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.371946 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 19:01:15.372150 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.371995 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-8.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 19:01:15.372755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.372742 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 19:01:15.372795 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.372757 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 19:01:15.372795 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.372766 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:01:15.374375 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.374358 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 19:01:15.375555 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.375538 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 19:01:15.378627 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.378608 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:01:15.378778 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.378768 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 19:01:15.378819 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.378811 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 07 19:01:15.378857 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.378827 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 19:01:15.378857 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.378841 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 19:01:15.378857 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.378851 2559 kubelet.go:398] "Adding apiserver pod source" Jul 07 19:01:15.378957 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.378861 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 19:01:15.379397 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.379381 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:01:15.379481 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.379400 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 19:01:15.382127 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.382088 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 19:01:15.384923 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.384899 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 19:01:15.385031 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.384941 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 19:01:15.386264 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386232 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386270 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386281 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386290 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386302 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386311 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386323 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386331 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386342 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 19:01:15.386349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386351 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 19:01:15.386622 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386365 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 19:01:15.386622 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.386374 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 19:01:15.387311 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.387298 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 19:01:15.387382 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.387314 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 19:01:15.390979 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.390959 2559 server.go:1268] "Started kubelet" Jul 07 19:01:15.391118 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.391086 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 19:01:15.391619 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.391157 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 19:01:15.391619 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.391286 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 19:01:15.392198 ip-10-0-137-8 systemd[1]: Started Kubernetes Kubelet. Jul 07 19:01:15.395087 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.394750 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 19:01:15.395180 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.395156 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f9g6p" Jul 07 19:01:15.395234 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.395185 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 19:01:15.395441 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.395402 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-8.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 19:01:15.395527 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.395491 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-8.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 19:01:15.396300 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.396279 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 07 19:01:15.398446 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.398411 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 19:01:15.400186 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.398561 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-8.ec2.internal.18c0171f3e092651 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-8.ec2.internal,UID:ip-10-0-137-8.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-8.ec2.internal,},FirstTimestamp:2026-07-07 19:01:15.390928465 +0000 UTC m=+0.412578460,LastTimestamp:2026-07-07 19:01:15.390928465 +0000 UTC m=+0.412578460,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-8.ec2.internal,}" Jul 07 19:01:15.400521 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.400505 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 19:01:15.400563 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.400524 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 19:01:15.400610 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.400588 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:15.401124 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401110 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 19:01:15.401262 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401196 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 19:01:15.401262 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401264 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 19:01:15.401427 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401347 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 07 19:01:15.401427 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401355 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 07 19:01:15.401427 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401419 2559 factory.go:55] Registering systemd factory Jul 07 19:01:15.401583 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401438 2559 factory.go:223] Registration of the systemd container factory successfully Jul 07 19:01:15.401673 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401657 2559 factory.go:153] Registering CRI-O factory Jul 07 19:01:15.401717 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401678 2559 factory.go:223] Registration of the crio container factory successfully Jul 07 19:01:15.401753 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401734 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 19:01:15.401788 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401758 2559 factory.go:103] Registering Raw factory Jul 07 19:01:15.401788 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.401776 2559 manager.go:1196] Started watching for new ooms in manager Jul 07 19:01:15.402212 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.402198 2559 manager.go:349] Starting recovery of all containers Jul 07 19:01:15.403882 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.403847 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f9g6p" Jul 07 19:01:15.412872 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.412664 2559 manager.go:354] Recovery completed Jul 07 19:01:15.413759 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.413742 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:01:15.414461 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.414440 2559 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 07 19:01:15.418221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.418202 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:01:15.419374 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.419354 2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-8.ec2.internal\" not found" node="ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.420660 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.420637 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:01:15.420726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.420683 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:01:15.420726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.420701 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:01:15.421348 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.421318 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 19:01:15.421428 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.421348 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 19:01:15.421428 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.421355 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:01:15.421428 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.421372 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 07 19:01:15.422137 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.422108 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:01:15.422246 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.422141 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:01:15.422246 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.422157 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:01:15.422473 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.422444 2559 policy_none.go:49] "None policy: Start" Jul 07 19:01:15.422473 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.422461 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 19:01:15.422473 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.422471 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 19:01:15.423675 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.423663 2559 policy_none.go:47] "Start" Jul 07 19:01:15.460139 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.460113 2559 manager.go:337] "Starting Device Plugin manager" Jul 07 19:01:15.460322 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.460172 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 19:01:15.460322 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.460206 2559 server.go:87] "Starting device plugin registration server" Jul 07 19:01:15.460588 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.460572 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 19:01:15.460652 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.460588 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 19:01:15.460716 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.460696 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 19:01:15.460858 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.460803 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 19:01:15.460858 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.460833 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 19:01:15.461481 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.461459 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 19:01:15.461585 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.461520 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:15.560741 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.560695 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:01:15.562117 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.562100 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:01:15.562211 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.562133 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:01:15.562211 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.562146 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:01:15.562211 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.562173 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.581834 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.581810 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.581925 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.581844 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-8.ec2.internal\": node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:15.588772 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.588743 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 19:01:15.590058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.590033 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 19:01:15.590180 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.590077 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 19:01:15.590180 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.590103 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 19:01:15.590180 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.590135 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 19:01:15.593161 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.593139 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:01:15.600470 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.600444 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:15.696360 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.696256 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal"] Jul 07 19:01:15.696360 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.696361 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:01:15.697568 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.697547 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:01:15.697690 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.697584 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:01:15.697690 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.697595 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:01:15.699571 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.699556 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:01:15.699684 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.699669 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.699784 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.699702 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:01:15.700524 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.700498 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:15.700524 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.700517 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:01:15.700693 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.700546 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:01:15.700693 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.700556 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:01:15.700693 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.700519 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:01:15.700693 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.700622 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:01:15.700693 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.700633 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:01:15.702070 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.702053 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.702150 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.702101 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 19:01:15.703039 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.703024 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientMemory" Jul 07 19:01:15.703115 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.703057 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 19:01:15.703115 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.703071 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeHasSufficientPID" Jul 07 19:01:15.703634 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.703620 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/178de8741949e5ed2708ec156f89e176-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"178de8741949e5ed2708ec156f89e176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.703677 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.703642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/178de8741949e5ed2708ec156f89e176-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"178de8741949e5ed2708ec156f89e176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.703677 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.703667 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ca19b4fae04ee4ffd4ef9397655850d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-8.ec2.internal\" (UID: \"2ca19b4fae04ee4ffd4ef9397655850d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.720103 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.720071 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-8.ec2.internal\" not found" node="ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.724591 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.724566 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-8.ec2.internal\" not found" node="ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.801722 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.801689 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:15.803992 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.803970 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/178de8741949e5ed2708ec156f89e176-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"178de8741949e5ed2708ec156f89e176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.804067 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.804001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/178de8741949e5ed2708ec156f89e176-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"178de8741949e5ed2708ec156f89e176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.804067 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.804019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ca19b4fae04ee4ffd4ef9397655850d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-8.ec2.internal\" (UID: \"2ca19b4fae04ee4ffd4ef9397655850d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.804067 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.804061 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2ca19b4fae04ee4ffd4ef9397655850d-config\") pod \"kube-apiserver-proxy-ip-10-0-137-8.ec2.internal\" (UID: \"2ca19b4fae04ee4ffd4ef9397655850d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.804159 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.804084 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/178de8741949e5ed2708ec156f89e176-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"178de8741949e5ed2708ec156f89e176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.804159 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:15.804090 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/178de8741949e5ed2708ec156f89e176-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal\" (UID: \"178de8741949e5ed2708ec156f89e176\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:15.901855 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:15.901814 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.003531 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.003434 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.023279 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.023244 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:16.028875 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.028532 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 07 19:01:16.104588 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.104553 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.205247 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.205202 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.308588 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.308510 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.339009 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.338976 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 19:01:16.339629 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.339141 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 19:01:16.339629 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.339192 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 19:01:16.407153 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.407105 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 18:56:15 +0000 UTC" deadline="2028-04-20 10:07:04.521942848 +0000 UTC" Jul 07 19:01:16.407153 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.407144 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15663h5m48.11480243s" Jul 07 19:01:16.409449 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.409427 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.509906 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.509866 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.611076 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.610961 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.711687 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.711648 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.779489 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:16.779448 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178de8741949e5ed2708ec156f89e176.slice/crio-807db3039f64b9ff1b68d506c16630ec8d14e1ebca5fe16c86c52bf5a8f447f3 WatchSource:0}: Error finding container 807db3039f64b9ff1b68d506c16630ec8d14e1ebca5fe16c86c52bf5a8f447f3: Status 404 returned error can't find the container with id 807db3039f64b9ff1b68d506c16630ec8d14e1ebca5fe16c86c52bf5a8f447f3 Jul 07 19:01:16.779958 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:16.779941 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca19b4fae04ee4ffd4ef9397655850d.slice/crio-0320ad77c76af7ce836196b7300929719936c81e94ccdd9bb41632117344426f WatchSource:0}: Error finding container 0320ad77c76af7ce836196b7300929719936c81e94ccdd9bb41632117344426f: Status 404 returned error can't find the container with id 0320ad77c76af7ce836196b7300929719936c81e94ccdd9bb41632117344426f Jul 07 19:01:16.786641 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.786616 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:01:16.812813 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.812774 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.912389 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.912313 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:01:16.913761 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:16.913733 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-8.ec2.internal\" not found" Jul 07 19:01:16.972074 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:16.972041 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:01:17.002138 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.002107 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" Jul 07 19:01:17.014242 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.014212 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:01:17.015224 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.015202 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" Jul 07 19:01:17.025720 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.025694 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 19:01:17.191117 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.191022 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:01:17.380161 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.380122 2559 apiserver.go:52] "Watching apiserver" Jul 07 19:01:17.389671 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.389635 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 19:01:17.393685 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.393626 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qss78","openshift-network-diagnostics/network-check-target-h8sbv","kube-system/konnectivity-agent-sdhn8","openshift-cluster-node-tuning-operator/tuned-mslnf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal","openshift-multus/multus-nw7wt","openshift-network-operator/iptables-alerter-rsnr7","openshift-ovn-kubernetes/ovnkube-node-tbbkn","kube-system/global-pull-secret-syncer-qfr4x","kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr","openshift-dns/node-resolver-6phhb","openshift-image-registry/node-ca-9ltm4","openshift-multus/multus-additional-cni-plugins-fd8kj"] Jul 07 19:01:17.395752 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.395715 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.398975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.397162 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.398975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.398259 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jkfsv\"" Jul 07 19:01:17.398975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.398593 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 19:01:17.398975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.398844 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:01:17.399311 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.399163 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.399664 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.399468 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 19:01:17.400605 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.399994 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 19:01:17.400605 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.400085 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 19:01:17.400605 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.400250 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gfwbb\"" Jul 07 19:01:17.400605 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.400444 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 19:01:17.400605 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.400524 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.401574 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.401552 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 19:01:17.401758 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.401613 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 19:01:17.402002 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.401983 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 19:01:17.402142 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.402123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.402324 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.402303 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gsnct\"" Jul 07 19:01:17.402972 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.402822 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6tc45\"" Jul 07 19:01:17.402972 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.402879 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:01:17.403137 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.402985 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 19:01:17.403800 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.403500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.405313 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.404585 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 19:01:17.405313 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.404807 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-w5khf\"" Jul 07 19:01:17.405313 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.404950 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 19:01:17.405313 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.405061 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 19:01:17.405313 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.405160 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.405313 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.405224 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:17.405813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.405680 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 19:01:17.405950 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.405891 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-clw46\"" Jul 07 19:01:17.405950 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.405924 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 19:01:17.406201 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.406170 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 19:01:17.406201 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.406193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.407802 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.407695 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:17.407802 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.407761 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:17.408969 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.408814 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 19:01:17.408969 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.408848 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 19:01:17.408969 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.408907 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 19:01:17.409492 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.409311 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-52v5m\"" Jul 07 19:01:17.409591 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.409567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.411273 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.411253 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:17.411401 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.411351 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:17.412461 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.412430 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 19:01:17.413075 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.412684 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 19:01:17.413587 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.413273 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 19:01:17.413587 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.413423 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.413587 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.413449 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 19:01:17.413587 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.413528 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 19:01:17.413812 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.413666 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 19:01:17.414128 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.414108 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kr9d\"" Jul 07 19:01:17.416925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.416901 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d9skn\"" Jul 07 19:01:17.417023 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.416936 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 19:01:17.417023 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.416947 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 19:01:17.417167 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/766beaa1-3925-48f3-8df8-49b8ab33a582-tmp\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.417266 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417186 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.417329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-kubernetes\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.417329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417293 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzfr5\" (UniqueName: \"kubernetes.io/projected/766beaa1-3925-48f3-8df8-49b8ab33a582-kube-api-access-qzfr5\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.417440 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.417440 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfc9r\" (UniqueName: \"kubernetes.io/projected/02cb4829-2b04-4397-90ba-40ce3ba73d9b-kube-api-access-wfc9r\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.417540 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84mgz\" (UniqueName: \"kubernetes.io/projected/009a0be4-4ea5-4254-a600-3f2b9953ff84-kube-api-access-84mgz\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.417540 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417469 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-socket-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.417540 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417502 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-sys\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.417702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-tuned\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.417702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.417702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cnibin\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.417702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-os-release\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.417898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417710 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-var-lib-kubelet\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.417898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-sys-fs\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.417898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/009a0be4-4ea5-4254-a600-3f2b9953ff84-iptables-alerter-script\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.417898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/009a0be4-4ea5-4254-a600-3f2b9953ff84-host-slash\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.417898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-registration-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.417898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417865 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysctl-conf\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.417898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417889 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-lib-modules\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417939 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d48bec1-d448-457f-922e-e75c8de4a3a4-agent-certs\") pod \"konnectivity-agent-sdhn8\" (UID: \"7d48bec1-d448-457f-922e-e75c8de4a3a4\") " pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysconfig\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.417995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-systemd\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-run\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418038 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418045 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418093 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjkx2\" (UniqueName: \"kubernetes.io/projected/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-kube-api-access-vjkx2\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418128 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-device-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-modprobe-d\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysctl-d\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d48bec1-d448-457f-922e-e75c8de4a3a4-konnectivity-ca\") pod \"konnectivity-agent-sdhn8\" (UID: \"7d48bec1-d448-457f-922e-e75c8de4a3a4\") " pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.418275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-host\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.418903 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-system-cni-dir\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.418903 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.418334 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.447426 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.447392 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdqjg" Jul 07 19:01:17.457317 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.457281 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdqjg" Jul 07 19:01:17.503031 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.502995 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 19:01:17.519042 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.518999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce283c41-58d4-4fe9-af91-4a005dd13dcb-host\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.519259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-multus-certs\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.519259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d48bec1-d448-457f-922e-e75c8de4a3a4-agent-certs\") pod \"konnectivity-agent-sdhn8\" (UID: \"7d48bec1-d448-457f-922e-e75c8de4a3a4\") " pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.519259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-systemd\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.519259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.519259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjkx2\" (UniqueName: \"kubernetes.io/projected/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-kube-api-access-vjkx2\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.519259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-k8s-cni-cncf-io\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.519259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxt2\" (UniqueName: \"kubernetes.io/projected/c3abdc92-0c27-4a09-9da5-21793e7289fb-kube-api-access-xsxt2\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519264 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-systemd\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519328 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-run-netns\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519363 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-device-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysctl-d\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-socket-dir-parent\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519464 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519492 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-system-cni-dir\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519576 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce283c41-58d4-4fe9-af91-4a005dd13dcb-serviceca\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.519611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovn-node-metrics-cert\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx85g\" (UniqueName: \"kubernetes.io/projected/f399a49f-d298-40ef-97b5-8dff9127f599-kube-api-access-zx85g\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519673 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-kubernetes\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-netns\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-cni-bin\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-daemon-config\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-systemd-units\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-var-lib-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519937 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f9aebed-5381-4ead-9371-161f4168b64a-dbus\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.519941 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520044 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-kubelet\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520071 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-kubernetes\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.520098 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-cni-bin\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520122 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-env-overrides\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520147 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-etc-selinux\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520149 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysctl-d\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84mgz\" (UniqueName: \"kubernetes.io/projected/009a0be4-4ea5-4254-a600-3f2b9953ff84-kube-api-access-84mgz\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520200 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-system-cni-dir\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-system-cni-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-kubelet\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520288 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-device-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-slash\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-log-socket\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovnkube-config\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cnibin\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-sys-fs\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520580 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cnibin\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/009a0be4-4ea5-4254-a600-3f2b9953ff84-iptables-alerter-script\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.520885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520688 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-sys-fs\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-cnibin\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-os-release\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3abdc92-0c27-4a09-9da5-21793e7289fb-cni-binary-copy\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520820 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8bp\" (UniqueName: \"kubernetes.io/projected/ce283c41-58d4-4fe9-af91-4a005dd13dcb-kube-api-access-vj8bp\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520860 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-cni-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysconfig\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-run\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-hostroot\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.520983 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-conf-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521008 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521010 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysconfig\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-modprobe-d\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521065 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-run\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-cni-netd\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f399a49f-d298-40ef-97b5-8dff9127f599-tmp-dir\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d48bec1-d448-457f-922e-e75c8de4a3a4-konnectivity-ca\") pod \"konnectivity-agent-sdhn8\" (UID: \"7d48bec1-d448-457f-922e-e75c8de4a3a4\") " pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-host\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.521644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521229 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-modprobe-d\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521240 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/009a0be4-4ea5-4254-a600-3f2b9953ff84-iptables-alerter-script\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521287 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f9aebed-5381-4ead-9371-161f4168b64a-kubelet-config\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521302 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-host\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521309 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-ovn\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521332 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/766beaa1-3925-48f3-8df8-49b8ab33a582-tmp\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521349 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzfr5\" (UniqueName: \"kubernetes.io/projected/766beaa1-3925-48f3-8df8-49b8ab33a582-kube-api-access-qzfr5\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521373 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-etc-kubernetes\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-node-log\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovnkube-script-lib\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f399a49f-d298-40ef-97b5-8dff9127f599-hosts-file\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-systemd\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfc9r\" (UniqueName: \"kubernetes.io/projected/02cb4829-2b04-4397-90ba-40ce3ba73d9b-kube-api-access-wfc9r\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-socket-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521585 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-sys\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.522539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-tuned\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521711 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxpx\" (UniqueName: \"kubernetes.io/projected/27b92afb-e567-44a1-9cc2-6482c2c6d934-kube-api-access-srxpx\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521716 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-os-release\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-cni-multus\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521774 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521812 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-sys\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-etc-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521872 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97m7f\" (UniqueName: \"kubernetes.io/projected/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-kube-api-access-97m7f\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-var-lib-kubelet\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521909 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-os-release\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/009a0be4-4ea5-4254-a600-3f2b9953ff84-host-slash\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-registration-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysctl-conf\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-lib-modules\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522053 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.523398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522277 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.521842 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d48bec1-d448-457f-922e-e75c8de4a3a4-konnectivity-ca\") pod \"konnectivity-agent-sdhn8\" (UID: \"7d48bec1-d448-457f-922e-e75c8de4a3a4\") " pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522309 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522315 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-socket-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522352 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-var-lib-kubelet\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522359 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02cb4829-2b04-4397-90ba-40ce3ba73d9b-registration-dir\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522401 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/009a0be4-4ea5-4254-a600-3f2b9953ff84-host-slash\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522458 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-sysctl-conf\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.524221 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.522460 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/766beaa1-3925-48f3-8df8-49b8ab33a582-lib-modules\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.525457 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.525428 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/766beaa1-3925-48f3-8df8-49b8ab33a582-tmp\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.525940 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.525913 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d48bec1-d448-457f-922e-e75c8de4a3a4-agent-certs\") pod \"konnectivity-agent-sdhn8\" (UID: \"7d48bec1-d448-457f-922e-e75c8de4a3a4\") " pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.528529 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.528500 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/766beaa1-3925-48f3-8df8-49b8ab33a582-etc-tuned\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.531596 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.531466 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjkx2\" (UniqueName: \"kubernetes.io/projected/6ce92b35-8e8a-4f68-b302-ccab54f6a7c8-kube-api-access-vjkx2\") pod \"multus-additional-cni-plugins-fd8kj\" (UID: \"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8\") " pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.531960 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.531938 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzfr5\" (UniqueName: \"kubernetes.io/projected/766beaa1-3925-48f3-8df8-49b8ab33a582-kube-api-access-qzfr5\") pod \"tuned-mslnf\" (UID: \"766beaa1-3925-48f3-8df8-49b8ab33a582\") " pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.533547 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.533382 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84mgz\" (UniqueName: \"kubernetes.io/projected/009a0be4-4ea5-4254-a600-3f2b9953ff84-kube-api-access-84mgz\") pod \"iptables-alerter-rsnr7\" (UID: \"009a0be4-4ea5-4254-a600-3f2b9953ff84\") " pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.533547 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.533524 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfc9r\" (UniqueName: \"kubernetes.io/projected/02cb4829-2b04-4397-90ba-40ce3ba73d9b-kube-api-access-wfc9r\") pod \"aws-ebs-csi-driver-node-z5vpr\" (UID: \"02cb4829-2b04-4397-90ba-40ce3ba73d9b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.597547 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.597472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" event={"ID":"178de8741949e5ed2708ec156f89e176","Type":"ContainerStarted","Data":"807db3039f64b9ff1b68d506c16630ec8d14e1ebca5fe16c86c52bf5a8f447f3"} Jul 07 19:01:17.599000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.598963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" event={"ID":"2ca19b4fae04ee4ffd4ef9397655850d","Type":"ContainerStarted","Data":"0320ad77c76af7ce836196b7300929719936c81e94ccdd9bb41632117344426f"} Jul 07 19:01:17.622478 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-system-cni-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.622478 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-kubelet\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.622755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622609 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-system-cni-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.622755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-slash\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.622755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-log-socket\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.622755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622711 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-slash\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.622755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622726 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovnkube-config\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-cnibin\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622795 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-log-socket\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-os-release\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622836 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3abdc92-0c27-4a09-9da5-21793e7289fb-cni-binary-copy\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8bp\" (UniqueName: \"kubernetes.io/projected/ce283c41-58d4-4fe9-af91-4a005dd13dcb-kube-api-access-vj8bp\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-cni-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.622914 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622959 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-hostroot\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622762 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-kubelet\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.622996 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret podName:3f9aebed-5381-4ead-9371-161f4168b64a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:18.122966786 +0000 UTC m=+3.144616773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret") pod "global-pull-secret-syncer-qfr4x" (UID: "3f9aebed-5381-4ead-9371-161f4168b64a") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.622915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-hostroot\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623029 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-cnibin\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-conf-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623101 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-os-release\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-cni-netd\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623164 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-cni-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f399a49f-d298-40ef-97b5-8dff9127f599-tmp-dir\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f9aebed-5381-4ead-9371-161f4168b64a-kubelet-config\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-ovn\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-etc-kubernetes\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-node-log\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623313 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovnkube-script-lib\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623337 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f399a49f-d298-40ef-97b5-8dff9127f599-hosts-file\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623365 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-systemd\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623401 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxpx\" (UniqueName: \"kubernetes.io/projected/27b92afb-e567-44a1-9cc2-6482c2c6d934-kube-api-access-srxpx\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-cni-multus\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-etc-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623506 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97m7f\" (UniqueName: \"kubernetes.io/projected/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-kube-api-access-97m7f\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.623659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623565 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-ovn\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623614 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-etc-kubernetes\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623616 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce283c41-58d4-4fe9-af91-4a005dd13dcb-host\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623644 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovnkube-config\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-multus-certs\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623664 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce283c41-58d4-4fe9-af91-4a005dd13dcb-host\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623690 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-multus-certs\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-k8s-cni-cncf-io\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623726 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxt2\" (UniqueName: \"kubernetes.io/projected/c3abdc92-0c27-4a09-9da5-21793e7289fb-kube-api-access-xsxt2\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623735 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-conf-dir\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.623740 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623770 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-cni-netd\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623775 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-node-log\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.623790 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs podName:27b92afb-e567-44a1-9cc2-6482c2c6d934 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:18.123772529 +0000 UTC m=+3.145422517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs") pod "network-metrics-daemon-qss78" (UID: "27b92afb-e567-44a1-9cc2-6482c2c6d934") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623720 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3abdc92-0c27-4a09-9da5-21793e7289fb-cni-binary-copy\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623514 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3f9aebed-5381-4ead-9371-161f4168b64a-kubelet-config\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623814 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623836 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-etc-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.624522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623842 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-run-netns\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-socket-dir-parent\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623895 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f399a49f-d298-40ef-97b5-8dff9127f599-hosts-file\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623959 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-socket-dir-parent\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624005 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-k8s-cni-cncf-io\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.623896 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624123 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce283c41-58d4-4fe9-af91-4a005dd13dcb-serviceca\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovn-node-metrics-cert\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx85g\" (UniqueName: \"kubernetes.io/projected/f399a49f-d298-40ef-97b5-8dff9127f599-kube-api-access-zx85g\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624165 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-netns\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624209 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-run-netns\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624210 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovnkube-script-lib\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624229 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-run-systemd\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624243 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-cni-multus\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624213 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-run-netns\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.625407 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624234 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-cni-bin\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624293 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624300 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3abdc92-0c27-4a09-9da5-21793e7289fb-host-var-lib-cni-bin\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624473 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f399a49f-d298-40ef-97b5-8dff9127f599-tmp-dir\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-daemon-config\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-systemd-units\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624759 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce283c41-58d4-4fe9-af91-4a005dd13dcb-serviceca\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624805 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-var-lib-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624832 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f9aebed-5381-4ead-9371-161f4168b64a-dbus\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-kubelet\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-cni-bin\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624934 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.624963 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-env-overrides\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.625006 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-var-lib-openvswitch\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.625014 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-cni-bin\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.625038 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3abdc92-0c27-4a09-9da5-21793e7289fb-multus-daemon-config\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.625049 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-host-kubelet\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.626233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.625086 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3f9aebed-5381-4ead-9371-161f4168b64a-dbus\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:17.627179 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.625184 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-systemd-units\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.627179 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.625384 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-env-overrides\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.628538 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.628491 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-ovn-node-metrics-cert\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.634213 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.634166 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:17.634213 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.634215 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:17.634510 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.634231 2559 projected.go:196] Error preparing data for projected volume kube-api-access-vpqdc for pod openshift-network-diagnostics/network-check-target-h8sbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:17.634510 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:17.634353 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc podName:3b955e71-d2a1-41f0-b9f6-431c57227ecc nodeName:}" failed. No retries permitted until 2026-07-07 19:01:18.134331239 +0000 UTC m=+3.155981244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vpqdc" (UniqueName: "kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc") pod "network-check-target-h8sbv" (UID: "3b955e71-d2a1-41f0-b9f6-431c57227ecc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:17.635184 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.635118 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8bp\" (UniqueName: \"kubernetes.io/projected/ce283c41-58d4-4fe9-af91-4a005dd13dcb-kube-api-access-vj8bp\") pod \"node-ca-9ltm4\" (UID: \"ce283c41-58d4-4fe9-af91-4a005dd13dcb\") " pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.637492 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.637462 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxt2\" (UniqueName: \"kubernetes.io/projected/c3abdc92-0c27-4a09-9da5-21793e7289fb-kube-api-access-xsxt2\") pod \"multus-nw7wt\" (UID: \"c3abdc92-0c27-4a09-9da5-21793e7289fb\") " pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.637986 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.637959 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx85g\" (UniqueName: \"kubernetes.io/projected/f399a49f-d298-40ef-97b5-8dff9127f599-kube-api-access-zx85g\") pod \"node-resolver-6phhb\" (UID: \"f399a49f-d298-40ef-97b5-8dff9127f599\") " pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.639037 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.639013 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97m7f\" (UniqueName: \"kubernetes.io/projected/d9e0d672-2ba3-4061-83d2-d7a12a98e1ff-kube-api-access-97m7f\") pod \"ovnkube-node-tbbkn\" (UID: \"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.641019 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.640963 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxpx\" (UniqueName: \"kubernetes.io/projected/27b92afb-e567-44a1-9cc2-6482c2c6d934-kube-api-access-srxpx\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:17.712008 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.711917 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsnr7" Jul 07 19:01:17.720798 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.720697 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" Jul 07 19:01:17.723657 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.723624 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009a0be4_4ea5_4254_a600_3f2b9953ff84.slice/crio-d7912f05e4fcbb9e49935dba7ced11140f5797b1a2f6a59a416fcf680b70db97 WatchSource:0}: Error finding container d7912f05e4fcbb9e49935dba7ced11140f5797b1a2f6a59a416fcf680b70db97: Status 404 returned error can't find the container with id d7912f05e4fcbb9e49935dba7ced11140f5797b1a2f6a59a416fcf680b70db97 Jul 07 19:01:17.731916 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.731800 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:17.733102 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.732949 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02cb4829_2b04_4397_90ba_40ce3ba73d9b.slice/crio-c84dee62860c31a1d5b600c256ce98d6e0cae2f6c31c2d8862593d098b309604 WatchSource:0}: Error finding container c84dee62860c31a1d5b600c256ce98d6e0cae2f6c31c2d8862593d098b309604: Status 404 returned error can't find the container with id c84dee62860c31a1d5b600c256ce98d6e0cae2f6c31c2d8862593d098b309604 Jul 07 19:01:17.741515 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.741281 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mslnf" Jul 07 19:01:17.743360 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.743325 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d48bec1_d448_457f_922e_e75c8de4a3a4.slice/crio-df3a627e5dc2247dc4dda4342cff7cf40443812d08a306acbc34f19d97056a7d WatchSource:0}: Error finding container df3a627e5dc2247dc4dda4342cff7cf40443812d08a306acbc34f19d97056a7d: Status 404 returned error can't find the container with id df3a627e5dc2247dc4dda4342cff7cf40443812d08a306acbc34f19d97056a7d Jul 07 19:01:17.749298 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.749260 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" Jul 07 19:01:17.751546 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.751204 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766beaa1_3925_48f3_8df8_49b8ab33a582.slice/crio-51ee545ef223dc1f7f047f20f83c64b6f8e4853a3156bca6e559894f1839ffbc WatchSource:0}: Error finding container 51ee545ef223dc1f7f047f20f83c64b6f8e4853a3156bca6e559894f1839ffbc: Status 404 returned error can't find the container with id 51ee545ef223dc1f7f047f20f83c64b6f8e4853a3156bca6e559894f1839ffbc Jul 07 19:01:17.759843 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.759626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nw7wt" Jul 07 19:01:17.760094 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.760050 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce92b35_8e8a_4f68_b302_ccab54f6a7c8.slice/crio-42c5797d9c26e0e68b93be1367f3a294bc4b957495127ecd4e3433c740762cfd WatchSource:0}: Error finding container 42c5797d9c26e0e68b93be1367f3a294bc4b957495127ecd4e3433c740762cfd: Status 404 returned error can't find the container with id 42c5797d9c26e0e68b93be1367f3a294bc4b957495127ecd4e3433c740762cfd Jul 07 19:01:17.768349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.768284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ltm4" Jul 07 19:01:17.770457 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.770420 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3abdc92_0c27_4a09_9da5_21793e7289fb.slice/crio-5a84246a8db9332a49ee80801addee87f8edbc13edf7d2ab612b0d599f91d61f WatchSource:0}: Error finding container 5a84246a8db9332a49ee80801addee87f8edbc13edf7d2ab612b0d599f91d61f: Status 404 returned error can't find the container with id 5a84246a8db9332a49ee80801addee87f8edbc13edf7d2ab612b0d599f91d61f Jul 07 19:01:17.777974 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.777759 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:17.779141 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.779108 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce283c41_58d4_4fe9_af91_4a005dd13dcb.slice/crio-e802b9eb38bd8ef95712a232ef333e5925fc1619fc4343bcbe87e015be19ac99 WatchSource:0}: Error finding container e802b9eb38bd8ef95712a232ef333e5925fc1619fc4343bcbe87e015be19ac99: Status 404 returned error can't find the container with id e802b9eb38bd8ef95712a232ef333e5925fc1619fc4343bcbe87e015be19ac99 Jul 07 19:01:17.783079 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.783049 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6phhb" Jul 07 19:01:17.789214 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.789179 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e0d672_2ba3_4061_83d2_d7a12a98e1ff.slice/crio-292398a6d8ccd6213c296fb0ed50b6a8c16d5785689b79816e4bb6a520ad7473 WatchSource:0}: Error finding container 292398a6d8ccd6213c296fb0ed50b6a8c16d5785689b79816e4bb6a520ad7473: Status 404 returned error can't find the container with id 292398a6d8ccd6213c296fb0ed50b6a8c16d5785689b79816e4bb6a520ad7473 Jul 07 19:01:17.796465 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:17.796420 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf399a49f_d298_40ef_97b5_8dff9127f599.slice/crio-8b001b3b58c460770ab26b733b092e55210a6e03a87f69130cc37f0639e4c3f3 WatchSource:0}: Error finding container 8b001b3b58c460770ab26b733b092e55210a6e03a87f69130cc37f0639e4c3f3: Status 404 returned error can't find the container with id 8b001b3b58c460770ab26b733b092e55210a6e03a87f69130cc37f0639e4c3f3 Jul 07 19:01:17.817993 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:17.817957 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 19:01:18.129137 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.129029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:18.129137 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.129085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:18.129401 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.129259 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:18.129401 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.129332 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs podName:27b92afb-e567-44a1-9cc2-6482c2c6d934 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:19.129312766 +0000 UTC m=+4.150962770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs") pod "network-metrics-daemon-qss78" (UID: "27b92afb-e567-44a1-9cc2-6482c2c6d934") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:18.129401 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.129262 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:18.129560 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.129430 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret podName:3f9aebed-5381-4ead-9371-161f4168b64a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:19.129410402 +0000 UTC m=+4.151060402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret") pod "global-pull-secret-syncer-qfr4x" (UID: "3f9aebed-5381-4ead-9371-161f4168b64a") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:18.229586 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.229545 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:18.229840 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.229728 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:18.229840 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.229762 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:18.229840 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.229779 2559 projected.go:196] Error preparing data for projected volume kube-api-access-vpqdc for pod openshift-network-diagnostics/network-check-target-h8sbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:18.229985 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.229848 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc podName:3b955e71-d2a1-41f0-b9f6-431c57227ecc nodeName:}" failed. No retries permitted until 2026-07-07 19:01:19.229830144 +0000 UTC m=+4.251480128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpqdc" (UniqueName: "kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc") pod "network-check-target-h8sbv" (UID: "3b955e71-d2a1-41f0-b9f6-431c57227ecc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:18.460460 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.460354 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:56:17 +0000 UTC" deadline="2028-02-14 12:00:47.039427225 +0000 UTC" Jul 07 19:01:18.460460 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.460397 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14080h59m28.579035152s" Jul 07 19:01:18.591307 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.590683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:18.591307 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:18.590861 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:18.603108 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.603061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6phhb" event={"ID":"f399a49f-d298-40ef-97b5-8dff9127f599","Type":"ContainerStarted","Data":"8b001b3b58c460770ab26b733b092e55210a6e03a87f69130cc37f0639e4c3f3"} Jul 07 19:01:18.605114 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.605070 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ltm4" event={"ID":"ce283c41-58d4-4fe9-af91-4a005dd13dcb","Type":"ContainerStarted","Data":"e802b9eb38bd8ef95712a232ef333e5925fc1619fc4343bcbe87e015be19ac99"} Jul 07 19:01:18.606742 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.606701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nw7wt" event={"ID":"c3abdc92-0c27-4a09-9da5-21793e7289fb","Type":"ContainerStarted","Data":"5a84246a8db9332a49ee80801addee87f8edbc13edf7d2ab612b0d599f91d61f"} Jul 07 19:01:18.608095 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.608032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerStarted","Data":"42c5797d9c26e0e68b93be1367f3a294bc4b957495127ecd4e3433c740762cfd"} Jul 07 19:01:18.609737 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.609681 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"292398a6d8ccd6213c296fb0ed50b6a8c16d5785689b79816e4bb6a520ad7473"} Jul 07 19:01:18.611596 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.611526 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mslnf" event={"ID":"766beaa1-3925-48f3-8df8-49b8ab33a582","Type":"ContainerStarted","Data":"51ee545ef223dc1f7f047f20f83c64b6f8e4853a3156bca6e559894f1839ffbc"} Jul 07 19:01:18.613532 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.613446 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdhn8" event={"ID":"7d48bec1-d448-457f-922e-e75c8de4a3a4","Type":"ContainerStarted","Data":"df3a627e5dc2247dc4dda4342cff7cf40443812d08a306acbc34f19d97056a7d"} Jul 07 19:01:18.615608 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.615513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" event={"ID":"02cb4829-2b04-4397-90ba-40ce3ba73d9b","Type":"ContainerStarted","Data":"c84dee62860c31a1d5b600c256ce98d6e0cae2f6c31c2d8862593d098b309604"} Jul 07 19:01:18.616952 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:18.616889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsnr7" event={"ID":"009a0be4-4ea5-4254-a600-3f2b9953ff84","Type":"ContainerStarted","Data":"d7912f05e4fcbb9e49935dba7ced11140f5797b1a2f6a59a416fcf680b70db97"} Jul 07 19:01:19.137192 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.137154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:19.137887 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.137412 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:19.137887 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.137498 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret podName:3f9aebed-5381-4ead-9371-161f4168b64a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:21.137477296 +0000 UTC m=+6.159127284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret") pod "global-pull-secret-syncer-qfr4x" (UID: "3f9aebed-5381-4ead-9371-161f4168b64a") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:19.137887 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.137412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:19.137887 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.137625 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:19.137887 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.137698 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs podName:27b92afb-e567-44a1-9cc2-6482c2c6d934 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:21.137679918 +0000 UTC m=+6.159329906 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs") pod "network-metrics-daemon-qss78" (UID: "27b92afb-e567-44a1-9cc2-6482c2c6d934") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:19.239350 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.238600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:19.239350 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.238844 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:19.239350 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.238866 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:19.239350 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.238879 2559 projected.go:196] Error preparing data for projected volume kube-api-access-vpqdc for pod openshift-network-diagnostics/network-check-target-h8sbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:19.239350 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.238943 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc podName:3b955e71-d2a1-41f0-b9f6-431c57227ecc nodeName:}" failed. No retries permitted until 2026-07-07 19:01:21.238923556 +0000 UTC m=+6.260573544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpqdc" (UniqueName: "kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc") pod "network-check-target-h8sbv" (UID: "3b955e71-d2a1-41f0-b9f6-431c57227ecc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:19.461503 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.461265 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 18:56:17 +0000 UTC" deadline="2028-01-27 18:05:47.942601568 +0000 UTC" Jul 07 19:01:19.461503 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.461332 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13655h4m28.481273335s" Jul 07 19:01:19.594560 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.593605 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:19.594560 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.593768 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:19.594560 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.594222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:19.594560 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:19.594348 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:19.635746 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:19.634997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" event={"ID":"2ca19b4fae04ee4ffd4ef9397655850d","Type":"ContainerStarted","Data":"a9152547b6aef77461bd07e5701f1354d231b0de378769f22e72f3846baee53e"} Jul 07 19:01:20.591266 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:20.590716 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:20.591266 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:20.590857 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:20.648073 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:20.647058 2559 generic.go:356] "Generic (PLEG): container finished" podID="178de8741949e5ed2708ec156f89e176" containerID="db62fccf1e53a50adc99664ae96761c5ac3504d4a4cb65e88e10d7e6c163ee85" exitCode=0 Jul 07 19:01:20.648073 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:20.647755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" event={"ID":"178de8741949e5ed2708ec156f89e176","Type":"ContainerDied","Data":"db62fccf1e53a50adc99664ae96761c5ac3504d4a4cb65e88e10d7e6c163ee85"} Jul 07 19:01:20.669654 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:20.669552 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-8.ec2.internal" podStartSLOduration=3.66952661 podStartE2EDuration="3.66952661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:19.654160081 +0000 UTC m=+4.675810088" watchObservedRunningTime="2026-07-07 19:01:20.66952661 +0000 UTC m=+5.691176619" Jul 07 19:01:21.158872 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:21.158029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:21.158872 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:21.158090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:21.158872 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.158258 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:21.158872 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.158325 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs podName:27b92afb-e567-44a1-9cc2-6482c2c6d934 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:25.158305608 +0000 UTC m=+10.179955595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs") pod "network-metrics-daemon-qss78" (UID: "27b92afb-e567-44a1-9cc2-6482c2c6d934") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:21.158872 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.158769 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:21.158872 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.158829 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret podName:3f9aebed-5381-4ead-9371-161f4168b64a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:25.158808608 +0000 UTC m=+10.180458598 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret") pod "global-pull-secret-syncer-qfr4x" (UID: "3f9aebed-5381-4ead-9371-161f4168b64a") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:21.258785 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:21.258739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:21.259009 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.258980 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:21.259087 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.259016 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:21.259087 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.259030 2559 projected.go:196] Error preparing data for projected volume kube-api-access-vpqdc for pod openshift-network-diagnostics/network-check-target-h8sbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:21.259200 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.259101 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc podName:3b955e71-d2a1-41f0-b9f6-431c57227ecc nodeName:}" failed. No retries permitted until 2026-07-07 19:01:25.259082477 +0000 UTC m=+10.280732467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpqdc" (UniqueName: "kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc") pod "network-check-target-h8sbv" (UID: "3b955e71-d2a1-41f0-b9f6-431c57227ecc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:21.591119 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:21.591034 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:21.591306 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.591203 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:21.591754 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:21.591576 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:21.591754 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:21.591684 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:22.590966 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:22.590928 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:22.591202 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:22.591063 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:23.591447 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:23.591365 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:23.591924 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:23.591534 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:23.591924 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:23.591564 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:23.594204 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:23.592481 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:24.590940 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:24.590902 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:24.591122 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:24.591043 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:24.659376 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:24.659304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" event={"ID":"178de8741949e5ed2708ec156f89e176","Type":"ContainerStarted","Data":"3018f3fd7ce2f6a8193b8e19888c573bf2c4eece42884bb7fe100432be9b3831"} Jul 07 19:01:24.677692 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:24.677631 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-8.ec2.internal" podStartSLOduration=7.677613829 podStartE2EDuration="7.677613829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:24.677231921 +0000 UTC m=+9.698881915" watchObservedRunningTime="2026-07-07 19:01:24.677613829 +0000 UTC m=+9.699263835" Jul 07 19:01:25.195284 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:25.195069 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:25.195284 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:25.195115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:25.195284 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.195263 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:25.195567 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.195263 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:25.195567 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.195331 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret podName:3f9aebed-5381-4ead-9371-161f4168b64a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.195311601 +0000 UTC m=+18.216961597 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret") pod "global-pull-secret-syncer-qfr4x" (UID: "3f9aebed-5381-4ead-9371-161f4168b64a") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:25.195567 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.195433 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs podName:27b92afb-e567-44a1-9cc2-6482c2c6d934 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.19540013 +0000 UTC m=+18.217050118 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs") pod "network-metrics-daemon-qss78" (UID: "27b92afb-e567-44a1-9cc2-6482c2c6d934") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:25.295766 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:25.295722 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:25.295933 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.295916 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:25.295999 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.295942 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:25.295999 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.295956 2559 projected.go:196] Error preparing data for projected volume kube-api-access-vpqdc for pod openshift-network-diagnostics/network-check-target-h8sbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:25.296067 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.296018 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc podName:3b955e71-d2a1-41f0-b9f6-431c57227ecc nodeName:}" failed. No retries permitted until 2026-07-07 19:01:33.295998998 +0000 UTC m=+18.317648983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpqdc" (UniqueName: "kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc") pod "network-check-target-h8sbv" (UID: "3b955e71-d2a1-41f0-b9f6-431c57227ecc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:25.591924 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:25.591839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:25.592085 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.591957 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:25.592085 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:25.592060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:25.592218 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:25.592152 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:26.590930 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:26.590466 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:26.590930 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:26.590631 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:27.590575 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:27.590535 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:27.590759 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:27.590540 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:27.590759 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:27.590671 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:27.590868 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:27.590756 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:28.591033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:28.590953 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:28.591499 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:28.591083 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:29.590735 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:29.590696 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:29.590928 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:29.590749 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:29.590928 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:29.590839 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:29.591031 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:29.590959 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:30.590670 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:30.590638 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:30.591084 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:30.590755 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:31.590796 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:31.590754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:31.590796 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:31.590798 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:31.591305 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:31.590920 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:31.591305 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:31.591049 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:32.590559 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:32.590482 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:32.590739 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:32.590608 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:33.255083 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:33.255044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:33.255083 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:33.255093 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:33.255687 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.255225 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:33.255687 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.255290 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret podName:3f9aebed-5381-4ead-9371-161f4168b64a nodeName:}" failed. No retries permitted until 2026-07-07 19:01:49.255273837 +0000 UTC m=+34.276923818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret") pod "global-pull-secret-syncer-qfr4x" (UID: "3f9aebed-5381-4ead-9371-161f4168b64a") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:33.255687 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.255225 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:33.255687 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.255419 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs podName:27b92afb-e567-44a1-9cc2-6482c2c6d934 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:49.255392716 +0000 UTC m=+34.277042698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs") pod "network-metrics-daemon-qss78" (UID: "27b92afb-e567-44a1-9cc2-6482c2c6d934") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:33.356468 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:33.356427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:33.356685 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.356645 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:33.356755 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.356694 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:33.356755 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.356708 2559 projected.go:196] Error preparing data for projected volume kube-api-access-vpqdc for pod openshift-network-diagnostics/network-check-target-h8sbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:33.356845 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.356783 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc podName:3b955e71-d2a1-41f0-b9f6-431c57227ecc nodeName:}" failed. No retries permitted until 2026-07-07 19:01:49.35676171 +0000 UTC m=+34.378411695 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpqdc" (UniqueName: "kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc") pod "network-check-target-h8sbv" (UID: "3b955e71-d2a1-41f0-b9f6-431c57227ecc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:33.591195 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:33.591069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:33.591195 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:33.591070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:33.591407 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.591238 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:33.591407 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:33.591331 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:34.590419 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:34.590386 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:34.590821 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:34.590498 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:35.591753 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:35.591699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:35.592210 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:35.591827 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:35.592210 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:35.591891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:35.592210 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:35.591983 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:36.590726 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:36.590683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:36.590899 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:36.590788 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:37.590708 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.590465 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:37.591379 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.590462 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:37.591379 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:37.590763 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:37.591379 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:37.590836 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:37.685836 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.685802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mslnf" event={"ID":"766beaa1-3925-48f3-8df8-49b8ab33a582","Type":"ContainerStarted","Data":"6045639674bec63ee2170b8d0875115dbbcaa365ea275f41761d9bc27bfc8159"} Jul 07 19:01:37.687481 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.687451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdhn8" event={"ID":"7d48bec1-d448-457f-922e-e75c8de4a3a4","Type":"ContainerStarted","Data":"923ebaa571f7c3024b485b8d8ccdde574187604f721f7c84f2dead7956d75f58"} Jul 07 19:01:37.688966 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.688941 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" event={"ID":"02cb4829-2b04-4397-90ba-40ce3ba73d9b","Type":"ContainerStarted","Data":"480643f35a1ac5bb82a782c1452d016ebc8a6109f8bcf6f4df3ecf0fcdcca9d4"} Jul 07 19:01:37.690297 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.690275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6phhb" event={"ID":"f399a49f-d298-40ef-97b5-8dff9127f599","Type":"ContainerStarted","Data":"8068834883d825765d0182e9e940a9fc0502efb3189c78075325a1c938c47446"} Jul 07 19:01:37.694702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.694673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ltm4" event={"ID":"ce283c41-58d4-4fe9-af91-4a005dd13dcb","Type":"ContainerStarted","Data":"d840fd0ef520f70c8aa7bd182bc9467bb49fe15927991df7cb9dcf843ee6b840"} Jul 07 19:01:37.695973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.695952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nw7wt" event={"ID":"c3abdc92-0c27-4a09-9da5-21793e7289fb","Type":"ContainerStarted","Data":"bce267a877d0abbd1d539a1e7c0746c97088e1fb062885a4f297d56a97a8294e"} Jul 07 19:01:37.697372 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.697349 2559 generic.go:356] "Generic (PLEG): container finished" podID="6ce92b35-8e8a-4f68-b302-ccab54f6a7c8" containerID="a4859b9e76795a7055171cfbd372661d3bc1d696ab35cca8ded34e1a192c6d55" exitCode=0 Jul 07 19:01:37.697468 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.697385 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerDied","Data":"a4859b9e76795a7055171cfbd372661d3bc1d696ab35cca8ded34e1a192c6d55"} Jul 07 19:01:37.740542 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.740489 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mslnf" podStartSLOduration=2.791109357 podStartE2EDuration="21.740472567s" totalImagesPullingTime="18.94936321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:37.71831392 +0000 UTC m=+22.739963924" watchObservedRunningTime="2026-07-07 19:01:37.740472567 +0000 UTC m=+22.762122570" Jul 07 19:01:37.740925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.740904 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6phhb" podStartSLOduration=2.83576991 podStartE2EDuration="21.740896422s" totalImagesPullingTime="18.905126512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:37.740386861 +0000 UTC m=+22.762036867" watchObservedRunningTime="2026-07-07 19:01:37.740896422 +0000 UTC m=+22.762546425" Jul 07 19:01:37.756413 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.756351 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9ltm4" podStartSLOduration=2.834559574 podStartE2EDuration="21.756328433s" totalImagesPullingTime="18.921768859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:37.755878045 +0000 UTC m=+22.777528050" watchObservedRunningTime="2026-07-07 19:01:37.756328433 +0000 UTC m=+22.777978438" Jul 07 19:01:37.773159 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.773109 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sdhn8" podStartSLOduration=2.816780174 podStartE2EDuration="21.773092307s" totalImagesPullingTime="18.956312133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:37.772511663 +0000 UTC m=+22.794161668" watchObservedRunningTime="2026-07-07 19:01:37.773092307 +0000 UTC m=+22.794742703" Jul 07 19:01:37.816467 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:37.816411 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nw7wt" podStartSLOduration=2.595568486 podStartE2EDuration="21.816395457s" totalImagesPullingTime="19.220826971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:37.816254339 +0000 UTC m=+22.837904345" watchObservedRunningTime="2026-07-07 19:01:37.816395457 +0000 UTC m=+22.838045461" Jul 07 19:01:38.590755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.590534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:38.591431 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:38.590845 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:38.701782 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.701751 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:01:38.702088 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.702065 2559 generic.go:356] "Generic (PLEG): container finished" podID="d9e0d672-2ba3-4061-83d2-d7a12a98e1ff" containerID="dfa1104a0c91ca8afb68f99736a23c723ae52d55341a677906406e18225c1b88" exitCode=1 Jul 07 19:01:38.702275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.702133 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"7ff320212974eff8cce3ed02689ead2d1166d8d597626c53543a8e3178afa840"} Jul 07 19:01:38.702275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.702228 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"a24c813f596875cc4dffa98d4df39d255474662a90a717ff67b1dc66f2fead20"} Jul 07 19:01:38.702275 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.702269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"c9a8b327b6f581858446110762a396354843600436e48da5a1804ea2add9008d"} Jul 07 19:01:38.702418 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.702296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerDied","Data":"dfa1104a0c91ca8afb68f99736a23c723ae52d55341a677906406e18225c1b88"} Jul 07 19:01:38.702418 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.702330 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"b7c932ee38593077184138b3e55ad0e19921b2ec7edc8120e22e49248202f7bc"} Jul 07 19:01:38.703488 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.703461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsnr7" event={"ID":"009a0be4-4ea5-4254-a600-3f2b9953ff84","Type":"ContainerStarted","Data":"7539fd3889c472c04d8d7d6dda5383856bab619a94a7e30e766300ad351b6f0d"} Jul 07 19:01:38.909251 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:38.909228 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 19:01:39.481231 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.481105 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T19:01:38.90924659Z","UUID":"b131a87d-5a3d-454c-a56b-bc2e0dedca0e","Handler":null,"Name":"","Endpoint":""} Jul 07 19:01:39.483077 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.483054 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 19:01:39.483246 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.483096 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 19:01:39.590544 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.590478 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:39.590544 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.590528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:39.590790 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:39.590634 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:39.590790 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:39.590756 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:39.710338 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.710295 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:01:39.710804 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.710771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"895eb478419aac1045ecbcee90a74b47d75050abf43af99d9a8e0179c2e2a917"} Jul 07 19:01:39.713634 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:39.713551 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" event={"ID":"02cb4829-2b04-4397-90ba-40ce3ba73d9b","Type":"ContainerStarted","Data":"466ab96a1d8775803d2af47a85bce899867ab76f90cdb6794b11f3bb505348b6"} Jul 07 19:01:40.591129 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:40.590881 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:40.591650 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:40.591260 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:40.718845 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:40.718754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" event={"ID":"02cb4829-2b04-4397-90ba-40ce3ba73d9b","Type":"ContainerStarted","Data":"18de493572b4750d4a13380094af5792dd249fe4d32e988f0791c1b3bc29f754"} Jul 07 19:01:40.746084 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:40.746021 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rsnr7" podStartSLOduration=5.770513684 podStartE2EDuration="24.746004064s" totalImagesPullingTime="18.97549038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:38.726489949 +0000 UTC m=+23.748139953" watchObservedRunningTime="2026-07-07 19:01:40.746004064 +0000 UTC m=+25.767654070" Jul 07 19:01:40.746297 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:40.746277 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z5vpr" podStartSLOduration=2.421683332 podStartE2EDuration="24.746271902s" totalImagesPullingTime="22.32458857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:40.746117356 +0000 UTC m=+25.767767374" watchObservedRunningTime="2026-07-07 19:01:40.746271902 +0000 UTC m=+25.767921906" Jul 07 19:01:41.590654 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:41.590619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:41.590817 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:41.590628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:41.590817 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:41.590736 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:41.590817 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:41.590796 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:41.725672 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:41.725643 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:01:41.726373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:41.726139 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"ac6e3f9ae38211e871035479b3ad6dbbd37adc37767cc086c60fd8012c4675c6"} Jul 07 19:01:42.064953 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:42.064826 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:42.065638 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:42.065602 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:42.590681 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:42.590650 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:42.590861 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:42.590767 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:43.590697 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.590671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:43.591273 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.590675 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:43.591273 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:43.590780 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:43.591273 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:43.590860 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:43.734301 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.734101 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:01:43.734604 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.734582 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"734f50ba11109574059728bb05201112011b0652eaa77ae7fcdb48c9cf9bca3d"} Jul 07 19:01:43.734926 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.734898 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:43.735027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.734978 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:43.735027 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.735026 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:43.735126 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.735111 2559 scope.go:117] "RemoveContainer" containerID="dfa1104a0c91ca8afb68f99736a23c723ae52d55341a677906406e18225c1b88" Jul 07 19:01:43.750769 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.750737 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:43.750926 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:43.750918 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:01:44.591225 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:44.591182 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:44.591689 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:44.591333 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:44.738419 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:44.738382 2559 generic.go:356] "Generic (PLEG): container finished" podID="6ce92b35-8e8a-4f68-b302-ccab54f6a7c8" containerID="38d61b4d2d95421c0acc5bb2b6bad9a972eb7f8b29c9b5b418efa42024f415af" exitCode=0 Jul 07 19:01:44.738580 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:44.738467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerDied","Data":"38d61b4d2d95421c0acc5bb2b6bad9a972eb7f8b29c9b5b418efa42024f415af"} Jul 07 19:01:44.742186 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:44.742149 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:01:44.742543 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:44.742520 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" event={"ID":"d9e0d672-2ba3-4061-83d2-d7a12a98e1ff","Type":"ContainerStarted","Data":"25b5b2e7d7893181f9ad376da0ffd38ae8c8b2a26a7a6efd297a4f08d1a9728d"} Jul 07 19:01:44.790261 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:44.790204 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" podStartSLOduration=8.64561108 podStartE2EDuration="28.790160619s" totalImagesPullingTime="20.144549539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:44.788605748 +0000 UTC m=+29.810255751" watchObservedRunningTime="2026-07-07 19:01:44.790160619 +0000 UTC m=+29.811810623" Jul 07 19:01:45.117447 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.117409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:45.117628 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.117550 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 19:01:45.118060 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.118038 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sdhn8" Jul 07 19:01:45.592091 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.591891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:45.592553 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.591955 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:45.592553 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:45.592205 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:45.592553 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:45.592240 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:45.620853 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.620817 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qfr4x"] Jul 07 19:01:45.624457 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.624422 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h8sbv"] Jul 07 19:01:45.624657 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.624543 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:45.624657 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:45.624638 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:45.625325 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.625304 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qss78"] Jul 07 19:01:45.745543 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.745473 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:45.745543 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:45.745494 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:45.745724 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:45.745706 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:45.746158 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:45.746138 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:46.749644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:46.749612 2559 generic.go:356] "Generic (PLEG): container finished" podID="6ce92b35-8e8a-4f68-b302-ccab54f6a7c8" containerID="c78d1621c421284d68085ce69bf880a32b526c74c60d06bddfc41ddb57426683" exitCode=0 Jul 07 19:01:46.750227 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:46.749654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerDied","Data":"c78d1621c421284d68085ce69bf880a32b526c74c60d06bddfc41ddb57426683"} Jul 07 19:01:47.590614 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:47.590574 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:47.590614 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:47.590619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:47.590807 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:47.590585 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:47.590807 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:47.590697 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:47.590807 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:47.590771 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:47.590892 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:47.590838 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:47.753943 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:47.753904 2559 generic.go:356] "Generic (PLEG): container finished" podID="6ce92b35-8e8a-4f68-b302-ccab54f6a7c8" containerID="4ae4c7d18d7ea2709a58448cd2e7514f68292a7d89c685d231aee459f2394574" exitCode=0 Jul 07 19:01:47.753943 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:47.753947 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerDied","Data":"4ae4c7d18d7ea2709a58448cd2e7514f68292a7d89c685d231aee459f2394574"} Jul 07 19:01:49.279300 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:49.279208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:49.279300 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:49.279261 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:49.279766 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.279368 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:49.279766 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.279377 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:49.279766 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.279441 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret podName:3f9aebed-5381-4ead-9371-161f4168b64a nodeName:}" failed. No retries permitted until 2026-07-07 19:02:21.279419564 +0000 UTC m=+66.301069549 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret") pod "global-pull-secret-syncer-qfr4x" (UID: "3f9aebed-5381-4ead-9371-161f4168b64a") : object "kube-system"/"original-pull-secret" not registered Jul 07 19:01:49.279766 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.279461 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs podName:27b92afb-e567-44a1-9cc2-6482c2c6d934 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:21.279450933 +0000 UTC m=+66.301100922 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs") pod "network-metrics-daemon-qss78" (UID: "27b92afb-e567-44a1-9cc2-6482c2c6d934") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 19:01:49.379552 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:49.379515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:49.379715 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.379697 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 19:01:49.379780 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.379719 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 19:01:49.379780 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.379728 2559 projected.go:196] Error preparing data for projected volume kube-api-access-vpqdc for pod openshift-network-diagnostics/network-check-target-h8sbv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:49.379857 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.379788 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc podName:3b955e71-d2a1-41f0-b9f6-431c57227ecc nodeName:}" failed. No retries permitted until 2026-07-07 19:02:21.379768549 +0000 UTC m=+66.401418536 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vpqdc" (UniqueName: "kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc") pod "network-check-target-h8sbv" (UID: "3b955e71-d2a1-41f0-b9f6-431c57227ecc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 19:01:49.591276 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:49.591185 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:49.591437 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:49.591188 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:49.591437 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.591292 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h8sbv" podUID="3b955e71-d2a1-41f0-b9f6-431c57227ecc" Jul 07 19:01:49.591437 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:49.591189 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:49.591437 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.591392 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qss78" podUID="27b92afb-e567-44a1-9cc2-6482c2c6d934" Jul 07 19:01:49.591584 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:49.591455 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qfr4x" podUID="3f9aebed-5381-4ead-9371-161f4168b64a" Jul 07 19:01:50.802323 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.802245 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-8.ec2.internal" event="NodeReady" Jul 07 19:01:50.802805 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.802394 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 19:01:50.853333 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.853229 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-slglq"] Jul 07 19:01:50.855699 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.855673 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cpnwl"] Jul 07 19:01:50.855855 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.855799 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:50.857740 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.857714 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:50.858851 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.858583 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 19:01:50.859005 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.858868 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 19:01:50.859671 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.859310 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 19:01:50.859671 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.859501 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bv84w\"" Jul 07 19:01:50.860982 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.860949 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 19:01:50.861202 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.861149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 19:01:50.861309 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.861226 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5hlmk\"" Jul 07 19:01:50.865991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.865961 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cpnwl"] Jul 07 19:01:50.868813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.868774 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slglq"] Jul 07 19:01:50.992680 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.992635 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77480f2d-0e6a-484f-acff-a713190d7e04-cert\") pod \"ingress-canary-slglq\" (UID: \"77480f2d-0e6a-484f-acff-a713190d7e04\") " pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:50.992856 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.992694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a3509-52f2-4f00-b988-b1402d0e916c-tmp-dir\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:50.992856 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.992726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0a3509-52f2-4f00-b988-b1402d0e916c-config-volume\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:50.992856 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.992753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8pzl\" (UniqueName: \"kubernetes.io/projected/fb0a3509-52f2-4f00-b988-b1402d0e916c-kube-api-access-c8pzl\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:50.992856 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.992844 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd88m\" (UniqueName: \"kubernetes.io/projected/77480f2d-0e6a-484f-acff-a713190d7e04-kube-api-access-gd88m\") pod \"ingress-canary-slglq\" (UID: \"77480f2d-0e6a-484f-acff-a713190d7e04\") " pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:50.993014 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:50.992885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.094029 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.093944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd88m\" (UniqueName: \"kubernetes.io/projected/77480f2d-0e6a-484f-acff-a713190d7e04-kube-api-access-gd88m\") pod \"ingress-canary-slglq\" (UID: \"77480f2d-0e6a-484f-acff-a713190d7e04\") " pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:51.094029 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.093981 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.094274 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.094038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77480f2d-0e6a-484f-acff-a713190d7e04-cert\") pod \"ingress-canary-slglq\" (UID: \"77480f2d-0e6a-484f-acff-a713190d7e04\") " pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:51.094274 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.094067 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a3509-52f2-4f00-b988-b1402d0e916c-tmp-dir\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.094274 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.094094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0a3509-52f2-4f00-b988-b1402d0e916c-config-volume\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.094274 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.094119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8pzl\" (UniqueName: \"kubernetes.io/projected/fb0a3509-52f2-4f00-b988-b1402d0e916c-kube-api-access-c8pzl\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.094274 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:51.094219 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:51.094554 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:51.094295 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls podName:fb0a3509-52f2-4f00-b988-b1402d0e916c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:51.594273551 +0000 UTC m=+36.615923537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls") pod "dns-default-cpnwl" (UID: "fb0a3509-52f2-4f00-b988-b1402d0e916c") : secret "dns-default-metrics-tls" not found Jul 07 19:01:51.094554 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.094464 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb0a3509-52f2-4f00-b988-b1402d0e916c-tmp-dir\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.094849 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.094826 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fb0a3509-52f2-4f00-b988-b1402d0e916c-config-volume\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.099202 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.099007 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/77480f2d-0e6a-484f-acff-a713190d7e04-cert\") pod \"ingress-canary-slglq\" (UID: \"77480f2d-0e6a-484f-acff-a713190d7e04\") " pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:51.102805 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.102739 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8pzl\" (UniqueName: \"kubernetes.io/projected/fb0a3509-52f2-4f00-b988-b1402d0e916c-kube-api-access-c8pzl\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.103561 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.103529 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd88m\" (UniqueName: \"kubernetes.io/projected/77480f2d-0e6a-484f-acff-a713190d7e04-kube-api-access-gd88m\") pod \"ingress-canary-slglq\" (UID: \"77480f2d-0e6a-484f-acff-a713190d7e04\") " pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:51.173308 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.173264 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-slglq" Jul 07 19:01:51.590561 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.590436 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:01:51.590561 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.590435 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:01:51.590561 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.590456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:01:51.594244 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.594207 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:01:51.594421 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.594248 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:01:51.594421 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.594352 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-546ff\"" Jul 07 19:01:51.594543 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.594517 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g87lc\"" Jul 07 19:01:51.594543 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.594525 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:01:51.594644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.594590 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:01:51.597813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:51.597788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:51.597980 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:51.597914 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:51.598045 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:51.597985 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls podName:fb0a3509-52f2-4f00-b988-b1402d0e916c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:52.597965571 +0000 UTC m=+37.619615554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls") pod "dns-default-cpnwl" (UID: "fb0a3509-52f2-4f00-b988-b1402d0e916c") : secret "dns-default-metrics-tls" not found Jul 07 19:01:52.223304 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.223269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-slglq"] Jul 07 19:01:52.228204 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:52.228148 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77480f2d_0e6a_484f_acff_a713190d7e04.slice/crio-1b34c5e792dfc0a03651e914ddeb057e97a42df92576abf9f2a0d32b3e2d944a WatchSource:0}: Error finding container 1b34c5e792dfc0a03651e914ddeb057e97a42df92576abf9f2a0d32b3e2d944a: Status 404 returned error can't find the container with id 1b34c5e792dfc0a03651e914ddeb057e97a42df92576abf9f2a0d32b3e2d944a Jul 07 19:01:52.500867 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.500836 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6"] Jul 07 19:01:52.502712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.502696 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" Jul 07 19:01:52.505374 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.505354 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 07 19:01:52.506997 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.506977 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 07 19:01:52.515233 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.515202 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jb2fn\"" Jul 07 19:01:52.535618 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.535587 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6"] Jul 07 19:01:52.605786 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.605692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:52.605786 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.605762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c-kube-api-access-th79w\") pod \"migrator-d57d58cc9-qpkf6\" (UID: \"ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" Jul 07 19:01:52.606020 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:52.605862 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:52.606020 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:52.605942 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls podName:fb0a3509-52f2-4f00-b988-b1402d0e916c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:54.60592044 +0000 UTC m=+39.627570428 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls") pod "dns-default-cpnwl" (UID: "fb0a3509-52f2-4f00-b988-b1402d0e916c") : secret "dns-default-metrics-tls" not found Jul 07 19:01:52.706519 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.706487 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c-kube-api-access-th79w\") pod \"migrator-d57d58cc9-qpkf6\" (UID: \"ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" Jul 07 19:01:52.715651 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.715610 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c-kube-api-access-th79w\") pod \"migrator-d57d58cc9-qpkf6\" (UID: \"ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" Jul 07 19:01:52.769979 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.769942 2559 generic.go:356] "Generic (PLEG): container finished" podID="6ce92b35-8e8a-4f68-b302-ccab54f6a7c8" containerID="15a0b093c17f21e2c6fdecd24a77bbc7c45c221a1494da715bf494c1df552390" exitCode=0 Jul 07 19:01:52.770191 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.770033 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerDied","Data":"15a0b093c17f21e2c6fdecd24a77bbc7c45c221a1494da715bf494c1df552390"} Jul 07 19:01:52.771248 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.771225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slglq" event={"ID":"77480f2d-0e6a-484f-acff-a713190d7e04","Type":"ContainerStarted","Data":"1b34c5e792dfc0a03651e914ddeb057e97a42df92576abf9f2a0d32b3e2d944a"} Jul 07 19:01:52.811648 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.811606 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" Jul 07 19:01:52.897533 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.897249 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fhjbt"] Jul 07 19:01:52.900754 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.900255 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:52.902940 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.902747 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 19:01:52.903093 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.903074 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 19:01:52.903316 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.903297 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 19:01:52.903420 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.903347 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 19:01:52.904260 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.903997 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f7xpr\"" Jul 07 19:01:52.916424 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.916389 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fhjbt"] Jul 07 19:01:52.979251 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:52.979212 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6"] Jul 07 19:01:52.982637 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:52.982604 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4d495e_7ff3_4c4b_a29a_c47e7e64e65c.slice/crio-268fcf0a2240a9535aa80f25c77d7d4ddb220d26bc39d7b9424fee7203fb0563 WatchSource:0}: Error finding container 268fcf0a2240a9535aa80f25c77d7d4ddb220d26bc39d7b9424fee7203fb0563: Status 404 returned error can't find the container with id 268fcf0a2240a9535aa80f25c77d7d4ddb220d26bc39d7b9424fee7203fb0563 Jul 07 19:01:53.009828 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.009791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c68e489b-4755-42fe-a811-119b84285d00-data-volume\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.010020 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.009840 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.010020 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.009893 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c68e489b-4755-42fe-a811-119b84285d00-crio-socket\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.010020 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.009945 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zn9l\" (UniqueName: \"kubernetes.io/projected/c68e489b-4755-42fe-a811-119b84285d00-kube-api-access-2zn9l\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.010020 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.009987 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.110310 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.110219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c68e489b-4755-42fe-a811-119b84285d00-data-volume\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.110310 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.110258 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.110310 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.110297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c68e489b-4755-42fe-a811-119b84285d00-crio-socket\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.110514 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.110329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zn9l\" (UniqueName: \"kubernetes.io/projected/c68e489b-4755-42fe-a811-119b84285d00-kube-api-access-2zn9l\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.110514 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.110355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.110514 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.110444 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:53.110620 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.110528 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:53.610511573 +0000 UTC m=+38.632161561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:53.110620 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.110452 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:53.110620 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.110583 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c68e489b-4755-42fe-a811-119b84285d00-crio-socket\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.110620 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.110599 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:53.610583086 +0000 UTC m=+38.632233082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : configmap "kube-rbac-proxy" not found Jul 07 19:01:53.110886 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.110866 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c68e489b-4755-42fe-a811-119b84285d00-data-volume\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.120493 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.120456 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zn9l\" (UniqueName: \"kubernetes.io/projected/c68e489b-4755-42fe-a811-119b84285d00-kube-api-access-2zn9l\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.614680 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.614641 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.615537 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.614718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:53.615537 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.614809 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:53.615537 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.614848 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:53.615537 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.614881 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:54.614860375 +0000 UTC m=+39.636510362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : configmap "kube-rbac-proxy" not found Jul 07 19:01:53.615537 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:53.614913 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:54.614895206 +0000 UTC m=+39.636545193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:53.778214 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.778154 2559 generic.go:356] "Generic (PLEG): container finished" podID="6ce92b35-8e8a-4f68-b302-ccab54f6a7c8" containerID="8dddd1791e82013512b09b5760f28e75d3fa919d2f377a0204c576ffdf3f6d61" exitCode=0 Jul 07 19:01:53.778214 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.778206 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerDied","Data":"8dddd1791e82013512b09b5760f28e75d3fa919d2f377a0204c576ffdf3f6d61"} Jul 07 19:01:53.779822 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:53.779775 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" event={"ID":"ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c","Type":"ContainerStarted","Data":"268fcf0a2240a9535aa80f25c77d7d4ddb220d26bc39d7b9424fee7203fb0563"} Jul 07 19:01:54.624258 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:54.624216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:54.624347 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:54.624381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:54.624429 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:54.624525 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:56.624505441 +0000 UTC m=+41.646155423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : configmap "kube-rbac-proxy" not found Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:54.624562 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:54.624616 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:01:56.624599945 +0000 UTC m=+41.646249932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:54.624615 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:54.624760 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:54.624648 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls podName:fb0a3509-52f2-4f00-b988-b1402d0e916c nodeName:}" failed. No retries permitted until 2026-07-07 19:01:58.624640034 +0000 UTC m=+43.646290020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls") pod "dns-default-cpnwl" (UID: "fb0a3509-52f2-4f00-b988-b1402d0e916c") : secret "dns-default-metrics-tls" not found Jul 07 19:01:54.787086 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:54.787041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" event={"ID":"6ce92b35-8e8a-4f68-b302-ccab54f6a7c8","Type":"ContainerStarted","Data":"4fa5958d8d9a49f71d159d4dede1351ca5457af6f30deddad79f3a44f75b369e"} Jul 07 19:01:54.816118 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:54.816054 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fd8kj" podStartSLOduration=7.108670951 podStartE2EDuration="38.816029763s" totalImagesPullingTime="31.707358812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:54.814316947 +0000 UTC m=+39.835966952" watchObservedRunningTime="2026-07-07 19:01:54.816029763 +0000 UTC m=+39.837679768" Jul 07 19:01:55.206214 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.206160 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fgf9l"] Jul 07 19:01:55.221278 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.221242 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fgf9l"] Jul 07 19:01:55.221447 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.221382 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.224156 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.224130 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 07 19:01:55.224330 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.224281 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nsgts\"" Jul 07 19:01:55.224502 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.224488 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 07 19:01:55.225262 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.225245 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 07 19:01:55.225339 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.225250 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 07 19:01:55.331469 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.331429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c37bf42-b23a-4c2b-94ca-90664c95c157-signing-key\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.331469 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.331477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c37bf42-b23a-4c2b-94ca-90664c95c157-signing-cabundle\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.331659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.331508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk8m\" (UniqueName: \"kubernetes.io/projected/2c37bf42-b23a-4c2b-94ca-90664c95c157-kube-api-access-5tk8m\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.432918 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.432811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c37bf42-b23a-4c2b-94ca-90664c95c157-signing-key\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.432918 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.432866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c37bf42-b23a-4c2b-94ca-90664c95c157-signing-cabundle\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.432918 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.432894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk8m\" (UniqueName: \"kubernetes.io/projected/2c37bf42-b23a-4c2b-94ca-90664c95c157-kube-api-access-5tk8m\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.433577 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.433553 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2c37bf42-b23a-4c2b-94ca-90664c95c157-signing-cabundle\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.435576 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.435552 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2c37bf42-b23a-4c2b-94ca-90664c95c157-signing-key\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.454974 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.454942 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk8m\" (UniqueName: \"kubernetes.io/projected/2c37bf42-b23a-4c2b-94ca-90664c95c157-kube-api-access-5tk8m\") pod \"service-ca-758f599c5-fgf9l\" (UID: \"2c37bf42-b23a-4c2b-94ca-90664c95c157\") " pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.531317 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.531283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-fgf9l" Jul 07 19:01:55.696935 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.696910 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6phhb_f399a49f-d298-40ef-97b5-8dff9127f599/dns-node-resolver/0.log" Jul 07 19:01:55.745154 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.745081 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fgf9l"] Jul 07 19:01:55.748503 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:01:55.748470 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c37bf42_b23a_4c2b_94ca_90664c95c157.slice/crio-9c0db35d01e58738fd19cd38cec48fe426e25ab121e2c53056b0ebd10465c44f WatchSource:0}: Error finding container 9c0db35d01e58738fd19cd38cec48fe426e25ab121e2c53056b0ebd10465c44f: Status 404 returned error can't find the container with id 9c0db35d01e58738fd19cd38cec48fe426e25ab121e2c53056b0ebd10465c44f Jul 07 19:01:55.792456 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.792404 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-slglq" event={"ID":"77480f2d-0e6a-484f-acff-a713190d7e04","Type":"ContainerStarted","Data":"52b2480d019e39089129fdeffe92c8a844e31450df5aff593d2df7a95ade2b36"} Jul 07 19:01:55.794083 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.794029 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" event={"ID":"ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c","Type":"ContainerStarted","Data":"cfa9ca1edf0a5af902af5776c44821eb3d16e68d83e51427cc3a4857646fc0bc"} Jul 07 19:01:55.795285 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.795258 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-fgf9l" event={"ID":"2c37bf42-b23a-4c2b-94ca-90664c95c157","Type":"ContainerStarted","Data":"9c0db35d01e58738fd19cd38cec48fe426e25ab121e2c53056b0ebd10465c44f"} Jul 07 19:01:55.811342 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:55.810859 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-slglq" podStartSLOduration=2.451195713 podStartE2EDuration="5.810830243s" totalImagesPullingTime="3.35963453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:55.809799345 +0000 UTC m=+40.831449349" watchObservedRunningTime="2026-07-07 19:01:55.810830243 +0000 UTC m=+40.832480248" Jul 07 19:01:56.641490 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:56.641442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:56.641691 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:56.641522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:56.641691 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:56.641670 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:01:56.641814 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:56.641740 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:00.641723562 +0000 UTC m=+45.663373545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : secret "insights-runtime-extractor-tls" not found Jul 07 19:01:56.659747 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:56.659709 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c68e489b-4755-42fe-a811-119b84285d00-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:01:56.800248 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:56.800204 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" event={"ID":"ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c","Type":"ContainerStarted","Data":"8b77286d9ecb0c685a1f634894026aaee3f6dd3bf52b235dced4b343093bf2cd"} Jul 07 19:01:56.818382 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:56.818319 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-qpkf6" podStartSLOduration=2.212097688 podStartE2EDuration="4.818301649s" totalImagesPullingTime="2.606203961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:56.817735042 +0000 UTC m=+41.839385047" watchObservedRunningTime="2026-07-07 19:01:56.818301649 +0000 UTC m=+41.839951653" Jul 07 19:01:56.881228 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:56.881153 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ltm4_ce283c41-58d4-4fe9-af91-4a005dd13dcb/node-ca/0.log" Jul 07 19:01:57.680509 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:57.680451 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-slglq_77480f2d-0e6a-484f-acff-a713190d7e04/serve-healthcheck-canary/0.log" Jul 07 19:01:57.881788 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:57.881742 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-qpkf6_ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c/migrator/0.log" Jul 07 19:01:58.080479 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:58.080398 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-qpkf6_ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c/graceful-termination/0.log" Jul 07 19:01:58.658501 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:58.658463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:01:58.658681 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:58.658629 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 07 19:01:58.658722 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:01:58.658698 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls podName:fb0a3509-52f2-4f00-b988-b1402d0e916c nodeName:}" failed. No retries permitted until 2026-07-07 19:02:06.658678365 +0000 UTC m=+51.680328346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls") pod "dns-default-cpnwl" (UID: "fb0a3509-52f2-4f00-b988-b1402d0e916c") : secret "dns-default-metrics-tls" not found Jul 07 19:01:59.809454 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:59.809417 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-fgf9l" event={"ID":"2c37bf42-b23a-4c2b-94ca-90664c95c157","Type":"ContainerStarted","Data":"713c8e8bc8eef0e95d1318adfaec1739809ea7a8b7d89b9c06cbb2b7aac2aef0"} Jul 07 19:01:59.826516 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:01:59.826454 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-fgf9l" podStartSLOduration=1.6500413329999999 podStartE2EDuration="4.826433289s" totalImagesPullingTime="3.176391956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:01:59.82635773 +0000 UTC m=+44.848007734" watchObservedRunningTime="2026-07-07 19:01:59.826433289 +0000 UTC m=+44.848083295" Jul 07 19:02:00.671820 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:00.671770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:02:00.671988 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:02:00.671920 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 07 19:02:00.672025 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:02:00.671993 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls podName:c68e489b-4755-42fe-a811-119b84285d00 nodeName:}" failed. No retries permitted until 2026-07-07 19:02:08.671974379 +0000 UTC m=+53.693624361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fhjbt" (UID: "c68e489b-4755-42fe-a811-119b84285d00") : secret "insights-runtime-extractor-tls" not found Jul 07 19:02:06.716390 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:06.716342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:02:06.718717 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:06.718693 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fb0a3509-52f2-4f00-b988-b1402d0e916c-metrics-tls\") pod \"dns-default-cpnwl\" (UID: \"fb0a3509-52f2-4f00-b988-b1402d0e916c\") " pod="openshift-dns/dns-default-cpnwl" Jul 07 19:02:06.779578 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:06.779542 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cpnwl" Jul 07 19:02:06.923861 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:06.923829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cpnwl"] Jul 07 19:02:06.927508 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:06.927477 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0a3509_52f2_4f00_b988_b1402d0e916c.slice/crio-2ce0b66eb53379687c9f5fb6b487df7bd833ad02fd68e9de76f3c64388b9c2ef WatchSource:0}: Error finding container 2ce0b66eb53379687c9f5fb6b487df7bd833ad02fd68e9de76f3c64388b9c2ef: Status 404 returned error can't find the container with id 2ce0b66eb53379687c9f5fb6b487df7bd833ad02fd68e9de76f3c64388b9c2ef Jul 07 19:02:07.832857 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:07.832813 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpnwl" event={"ID":"fb0a3509-52f2-4f00-b988-b1402d0e916c","Type":"ContainerStarted","Data":"2ce0b66eb53379687c9f5fb6b487df7bd833ad02fd68e9de76f3c64388b9c2ef"} Jul 07 19:02:08.732084 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:08.732043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:02:08.734452 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:08.734415 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c68e489b-4755-42fe-a811-119b84285d00-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fhjbt\" (UID: \"c68e489b-4755-42fe-a811-119b84285d00\") " pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:02:08.848697 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:08.848655 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fhjbt" Jul 07 19:02:09.108855 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:09.108825 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fhjbt"] Jul 07 19:02:09.112831 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:09.112803 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68e489b_4755_42fe_a811_119b84285d00.slice/crio-4b2f3e2141ec188671190a71a9ba5f6046ca966df5b0ed0d4536c62897be052f WatchSource:0}: Error finding container 4b2f3e2141ec188671190a71a9ba5f6046ca966df5b0ed0d4536c62897be052f: Status 404 returned error can't find the container with id 4b2f3e2141ec188671190a71a9ba5f6046ca966df5b0ed0d4536c62897be052f Jul 07 19:02:09.841527 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:09.841483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fhjbt" event={"ID":"c68e489b-4755-42fe-a811-119b84285d00","Type":"ContainerStarted","Data":"4b2f3e2141ec188671190a71a9ba5f6046ca966df5b0ed0d4536c62897be052f"} Jul 07 19:02:09.844524 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:09.844476 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpnwl" event={"ID":"fb0a3509-52f2-4f00-b988-b1402d0e916c","Type":"ContainerStarted","Data":"a620a0acbce079ccdbf710bce20240a8d0cd9d7c90e2aecd870c95b5504bee6b"} Jul 07 19:02:09.844687 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:09.844541 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cpnwl" event={"ID":"fb0a3509-52f2-4f00-b988-b1402d0e916c","Type":"ContainerStarted","Data":"b6766144de93ccdfa9b0baebdfa81c66482f0a7d667d355cab3a04da7c30268b"} Jul 07 19:02:09.844879 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:09.844763 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cpnwl" Jul 07 19:02:09.866060 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:09.865854 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cpnwl" podStartSLOduration=17.824321748 podStartE2EDuration="19.865828507s" totalImagesPullingTime="2.041506759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:09.864470858 +0000 UTC m=+54.886120863" watchObservedRunningTime="2026-07-07 19:02:09.865828507 +0000 UTC m=+54.887478512" Jul 07 19:02:13.859944 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:13.859906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fhjbt" event={"ID":"c68e489b-4755-42fe-a811-119b84285d00","Type":"ContainerStarted","Data":"0074c8b2b4e6cdf7a0a2c20ef2e746aa115abeaa5eb17a61895ae5914415aff7"} Jul 07 19:02:13.860345 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:13.859969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fhjbt" event={"ID":"c68e489b-4755-42fe-a811-119b84285d00","Type":"ContainerStarted","Data":"f36dac186ea9cdfe8ab9aa668bbea9ce2869ae38ce2608ef6cf13d5abb780535"} Jul 07 19:02:15.766538 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:15.766509 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tbbkn" Jul 07 19:02:16.871638 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:16.871606 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fhjbt" event={"ID":"c68e489b-4755-42fe-a811-119b84285d00","Type":"ContainerStarted","Data":"e0ea25e79c2754915063ef585be6c64b38b4e8c8ba0e0016f6ac984e7568314a"} Jul 07 19:02:16.892126 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:16.892074 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fhjbt" podStartSLOduration=17.402226411 podStartE2EDuration="24.892054133s" totalImagesPullingTime="7.489827722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:16.892035815 +0000 UTC m=+61.913685833" watchObservedRunningTime="2026-07-07 19:02:16.892054133 +0000 UTC m=+61.913704186" Jul 07 19:02:17.172229 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.172196 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb"] Jul 07 19:02:17.173906 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.173890 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.176681 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.176652 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 07 19:02:17.176824 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.176750 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 07 19:02:17.176824 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.176795 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 07 19:02:17.177030 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.177017 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 07 19:02:17.185604 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.185575 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c"] Jul 07 19:02:17.187451 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.187429 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.190581 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.190555 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 07 19:02:17.192025 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.192002 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 07 19:02:17.192155 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.192003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 07 19:02:17.192357 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.192340 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 07 19:02:17.193804 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.193779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb"] Jul 07 19:02:17.200820 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.200790 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e23edda3-f6d2-473b-b9d0-147023cfc65b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.200973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.200828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mgd\" (UniqueName: \"kubernetes.io/projected/e23edda3-f6d2-473b-b9d0-147023cfc65b-kube-api-access-74mgd\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.200973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.200944 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23edda3-f6d2-473b-b9d0-147023cfc65b-tmp\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.213635 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.213601 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c"] Jul 07 19:02:17.302002 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.301959 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e23edda3-f6d2-473b-b9d0-147023cfc65b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.302002 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.301995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mgd\" (UniqueName: \"kubernetes.io/projected/e23edda3-f6d2-473b-b9d0-147023cfc65b-kube-api-access-74mgd\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.302291 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302033 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.302291 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-ca\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.302291 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d297e9fa-01ed-4c3b-9588-464f8977116f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.302291 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23edda3-f6d2-473b-b9d0-147023cfc65b-tmp\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.302457 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302439 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmz6\" (UniqueName: \"kubernetes.io/projected/d297e9fa-01ed-4c3b-9588-464f8977116f-kube-api-access-7mmz6\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.302549 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-hub\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.302709 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.302755 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.302705 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23edda3-f6d2-473b-b9d0-147023cfc65b-tmp\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.305014 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.304983 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e23edda3-f6d2-473b-b9d0-147023cfc65b-klusterlet-config\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.317856 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.317810 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mgd\" (UniqueName: \"kubernetes.io/projected/e23edda3-f6d2-473b-b9d0-147023cfc65b-kube-api-access-74mgd\") pod \"klusterlet-addon-workmgr-8dcd764d7-brbfb\" (UID: \"e23edda3-f6d2-473b-b9d0-147023cfc65b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.403527 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.403483 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-hub\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.403527 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.403522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.403844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.403548 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.403844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.403565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-ca\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.403844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.403606 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d297e9fa-01ed-4c3b-9588-464f8977116f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.403844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.403632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmz6\" (UniqueName: \"kubernetes.io/projected/d297e9fa-01ed-4c3b-9588-464f8977116f-kube-api-access-7mmz6\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.404473 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.404443 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d297e9fa-01ed-4c3b-9588-464f8977116f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.406304 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.406278 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.406448 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.406384 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-hub\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.406448 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.406417 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.406564 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.406493 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d297e9fa-01ed-4c3b-9588-464f8977116f-ca\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.416497 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.416461 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmz6\" (UniqueName: \"kubernetes.io/projected/d297e9fa-01ed-4c3b-9588-464f8977116f-kube-api-access-7mmz6\") pod \"cluster-proxy-proxy-agent-86688c5f84-qkx9c\" (UID: \"d297e9fa-01ed-4c3b-9588-464f8977116f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.483242 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.483141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:17.506548 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.506517 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" Jul 07 19:02:17.639809 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.639647 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb"] Jul 07 19:02:17.642420 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:17.642389 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23edda3_f6d2_473b_b9d0_147023cfc65b.slice/crio-28cd99b82a57092f3e7325050bae141d0a28e8ae05ee7cf30ccac1561aec776c WatchSource:0}: Error finding container 28cd99b82a57092f3e7325050bae141d0a28e8ae05ee7cf30ccac1561aec776c: Status 404 returned error can't find the container with id 28cd99b82a57092f3e7325050bae141d0a28e8ae05ee7cf30ccac1561aec776c Jul 07 19:02:17.669239 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.666614 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c"] Jul 07 19:02:17.672452 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:17.672415 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd297e9fa_01ed_4c3b_9588_464f8977116f.slice/crio-acab36e24a7ad5538ac5c298ca40e799a1370bdb3dd8423d2139c33bbbc63412 WatchSource:0}: Error finding container acab36e24a7ad5538ac5c298ca40e799a1370bdb3dd8423d2139c33bbbc63412: Status 404 returned error can't find the container with id acab36e24a7ad5538ac5c298ca40e799a1370bdb3dd8423d2139c33bbbc63412 Jul 07 19:02:17.876265 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.876231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" event={"ID":"e23edda3-f6d2-473b-b9d0-147023cfc65b","Type":"ContainerStarted","Data":"28cd99b82a57092f3e7325050bae141d0a28e8ae05ee7cf30ccac1561aec776c"} Jul 07 19:02:17.877465 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:17.877440 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" event={"ID":"d297e9fa-01ed-4c3b-9588-464f8977116f","Type":"ContainerStarted","Data":"acab36e24a7ad5538ac5c298ca40e799a1370bdb3dd8423d2139c33bbbc63412"} Jul 07 19:02:19.851111 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:19.850408 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cpnwl" Jul 07 19:02:21.343305 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.343271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:02:21.343305 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.343313 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:02:21.345707 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.345678 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 19:02:21.346529 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.346502 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 19:02:21.357128 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.357095 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3f9aebed-5381-4ead-9371-161f4168b64a-original-pull-secret\") pod \"global-pull-secret-syncer-qfr4x\" (UID: \"3f9aebed-5381-4ead-9371-161f4168b64a\") " pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:02:21.357312 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.357152 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27b92afb-e567-44a1-9cc2-6482c2c6d934-metrics-certs\") pod \"network-metrics-daemon-qss78\" (UID: \"27b92afb-e567-44a1-9cc2-6482c2c6d934\") " pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:02:21.444037 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.443996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:02:21.446945 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.446910 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 19:02:21.457645 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.457606 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 19:02:21.469117 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.469085 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpqdc\" (UniqueName: \"kubernetes.io/projected/3b955e71-d2a1-41f0-b9f6-431c57227ecc-kube-api-access-vpqdc\") pod \"network-check-target-h8sbv\" (UID: \"3b955e71-d2a1-41f0-b9f6-431c57227ecc\") " pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:02:21.606060 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.605972 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-546ff\"" Jul 07 19:02:21.613243 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.613209 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g87lc\"" Jul 07 19:02:21.613422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.613311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:02:21.616443 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.616422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qfr4x" Jul 07 19:02:21.621195 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:21.621153 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qss78" Jul 07 19:02:23.254727 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.254649 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h8sbv"] Jul 07 19:02:23.260450 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:23.260354 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b955e71_d2a1_41f0_b9f6_431c57227ecc.slice/crio-7d3e53cb4dc470cd329ead7455a73e1ac6ef868bae95ca15f113a46ac8d13da4 WatchSource:0}: Error finding container 7d3e53cb4dc470cd329ead7455a73e1ac6ef868bae95ca15f113a46ac8d13da4: Status 404 returned error can't find the container with id 7d3e53cb4dc470cd329ead7455a73e1ac6ef868bae95ca15f113a46ac8d13da4 Jul 07 19:02:23.277689 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.277651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qfr4x"] Jul 07 19:02:23.282480 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:23.282436 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9aebed_5381_4ead_9371_161f4168b64a.slice/crio-1c2cc3117c623d95d6484780ae093285a373c026dec2395349aa8e46897f682c WatchSource:0}: Error finding container 1c2cc3117c623d95d6484780ae093285a373c026dec2395349aa8e46897f682c: Status 404 returned error can't find the container with id 1c2cc3117c623d95d6484780ae093285a373c026dec2395349aa8e46897f682c Jul 07 19:02:23.484511 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.484472 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qss78"] Jul 07 19:02:23.487863 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:23.487825 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b92afb_e567_44a1_9cc2_6482c2c6d934.slice/crio-435b5f3420ef290430024da97255dcd5ef4f25c1d253a0a31fa88e653a117b7e WatchSource:0}: Error finding container 435b5f3420ef290430024da97255dcd5ef4f25c1d253a0a31fa88e653a117b7e: Status 404 returned error can't find the container with id 435b5f3420ef290430024da97255dcd5ef4f25c1d253a0a31fa88e653a117b7e Jul 07 19:02:23.907638 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.907601 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" event={"ID":"d297e9fa-01ed-4c3b-9588-464f8977116f","Type":"ContainerStarted","Data":"bc0fffad2d6ce68d0d8d0fd799ed485ec66dfa2de478729d0dd582e89cf4b289"} Jul 07 19:02:23.907813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.907654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" event={"ID":"d297e9fa-01ed-4c3b-9588-464f8977116f","Type":"ContainerStarted","Data":"0a1981165857da1f5d313e6943cf716e9dc45ebbf5f132e4c67a817bcce6b9e1"} Jul 07 19:02:23.907813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.907672 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" event={"ID":"d297e9fa-01ed-4c3b-9588-464f8977116f","Type":"ContainerStarted","Data":"0126c19dac084a117bffff15138ee3a612da47a6277dad59bc9c6bd0f5e0442e"} Jul 07 19:02:23.909427 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.909388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qfr4x" event={"ID":"3f9aebed-5381-4ead-9371-161f4168b64a","Type":"ContainerStarted","Data":"1c2cc3117c623d95d6484780ae093285a373c026dec2395349aa8e46897f682c"} Jul 07 19:02:23.910991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.910929 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h8sbv" event={"ID":"3b955e71-d2a1-41f0-b9f6-431c57227ecc","Type":"ContainerStarted","Data":"7d3e53cb4dc470cd329ead7455a73e1ac6ef868bae95ca15f113a46ac8d13da4"} Jul 07 19:02:23.912793 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.912762 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" event={"ID":"e23edda3-f6d2-473b-b9d0-147023cfc65b","Type":"ContainerStarted","Data":"0b65afd4e10661b5097c3d1380b63f7074ebe71d414a4976f3b857d17f7a6545"} Jul 07 19:02:23.913414 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.913356 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:23.915205 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.915137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qss78" event={"ID":"27b92afb-e567-44a1-9cc2-6482c2c6d934","Type":"ContainerStarted","Data":"435b5f3420ef290430024da97255dcd5ef4f25c1d253a0a31fa88e653a117b7e"} Jul 07 19:02:23.915451 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.915415 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" Jul 07 19:02:23.931672 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.931601 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-86688c5f84-qkx9c" podStartSLOduration=1.487181003 podStartE2EDuration="6.931582384s" totalImagesPullingTime="5.444401381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:23.929746788 +0000 UTC m=+68.951396804" watchObservedRunningTime="2026-07-07 19:02:23.931582384 +0000 UTC m=+68.953232394" Jul 07 19:02:23.947575 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:23.947515 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8dcd764d7-brbfb" podStartSLOduration=1.493687854 podStartE2EDuration="6.947500565s" totalImagesPullingTime="5.453812711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:23.946292372 +0000 UTC m=+68.967942378" watchObservedRunningTime="2026-07-07 19:02:23.947500565 +0000 UTC m=+68.969150570" Jul 07 19:02:25.930008 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:25.929971 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qss78" event={"ID":"27b92afb-e567-44a1-9cc2-6482c2c6d934","Type":"ContainerStarted","Data":"2d0e7b7037a80a04b51d15427b5a570a8484d5a35d36023ef4951bf70121d738"} Jul 07 19:02:25.930522 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:25.930072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qss78" event={"ID":"27b92afb-e567-44a1-9cc2-6482c2c6d934","Type":"ContainerStarted","Data":"e03d59bd8b2698a79b5e88c04348be4c21fa971086e40265f0f407b535073163"} Jul 07 19:02:25.948415 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:25.948361 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qss78" podStartSLOduration=68.595318581 podStartE2EDuration="1m9.948343773s" totalImagesPullingTime="1.353025192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:25.947024483 +0000 UTC m=+70.968674488" watchObservedRunningTime="2026-07-07 19:02:25.948343773 +0000 UTC m=+70.969993776" Jul 07 19:02:28.942444 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:28.942393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h8sbv" event={"ID":"3b955e71-d2a1-41f0-b9f6-431c57227ecc","Type":"ContainerStarted","Data":"b6151905726418089dbc448e8200a1a69c787294e433c74e10ab4917dfbeddee"} Jul 07 19:02:28.942961 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:28.942472 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:02:28.943894 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:28.943869 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qfr4x" event={"ID":"3f9aebed-5381-4ead-9371-161f4168b64a","Type":"ContainerStarted","Data":"cc5e57f9a25f247df37dff99d9368f440a17812ca0c6d7a834afec639ce12ece"} Jul 07 19:02:28.961742 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:28.961686 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h8sbv" podStartSLOduration=68.316901157 podStartE2EDuration="1m12.96167114s" totalImagesPullingTime="4.644769983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:28.960875555 +0000 UTC m=+73.982525572" watchObservedRunningTime="2026-07-07 19:02:28.96167114 +0000 UTC m=+73.983321172" Jul 07 19:02:28.976907 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:28.976850 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qfr4x" podStartSLOduration=68.349871693 podStartE2EDuration="1m12.976833776s" totalImagesPullingTime="4.626962083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:28.975754386 +0000 UTC m=+73.997404391" watchObservedRunningTime="2026-07-07 19:02:28.976833776 +0000 UTC m=+73.998483780" Jul 07 19:02:31.634102 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.634065 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nd6pq"] Jul 07 19:02:31.637833 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.637803 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.640120 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.640086 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4kc7z\"" Jul 07 19:02:31.640924 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.640893 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 19:02:31.641354 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.641338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 19:02:31.641594 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.641441 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 19:02:31.641767 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.641749 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 19:02:31.641980 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.641886 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 19:02:31.642203 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.642125 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 19:02:31.723410 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723614 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723433 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-sys\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723614 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015fa7a7-c259-40ff-9404-c9d00bcb1e17-metrics-client-ca\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723614 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723614 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-textfile\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723816 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-wtmp\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723816 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723655 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-tls\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723816 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723699 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgx9f\" (UniqueName: \"kubernetes.io/projected/015fa7a7-c259-40ff-9404-c9d00bcb1e17-kube-api-access-wgx9f\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.723816 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.723724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-root\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824504 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824692 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-textfile\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824692 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-wtmp\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824792 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-tls\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824792 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824741 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgx9f\" (UniqueName: \"kubernetes.io/projected/015fa7a7-c259-40ff-9404-c9d00bcb1e17-kube-api-access-wgx9f\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824792 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-root\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824857 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-sys\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824854 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-wtmp\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015fa7a7-c259-40ff-9404-c9d00bcb1e17-metrics-client-ca\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.824925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824897 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-textfile\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.825153 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.824977 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-root\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.825153 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.825107 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/015fa7a7-c259-40ff-9404-c9d00bcb1e17-sys\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.825272 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.825245 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-accelerators-collector-config\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.825437 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.825403 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/015fa7a7-c259-40ff-9404-c9d00bcb1e17-metrics-client-ca\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.827384 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.827352 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-tls\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.827518 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.827503 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/015fa7a7-c259-40ff-9404-c9d00bcb1e17-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.839960 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.839923 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgx9f\" (UniqueName: \"kubernetes.io/projected/015fa7a7-c259-40ff-9404-c9d00bcb1e17-kube-api-access-wgx9f\") pod \"node-exporter-nd6pq\" (UID: \"015fa7a7-c259-40ff-9404-c9d00bcb1e17\") " pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.961000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:31.960889 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nd6pq" Jul 07 19:02:31.975179 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:31.975125 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015fa7a7_c259_40ff_9404_c9d00bcb1e17.slice/crio-cd9503f0a999258d815eacfe3225701dbbce46ad8ede51a49dd9126df5061503 WatchSource:0}: Error finding container cd9503f0a999258d815eacfe3225701dbbce46ad8ede51a49dd9126df5061503: Status 404 returned error can't find the container with id cd9503f0a999258d815eacfe3225701dbbce46ad8ede51a49dd9126df5061503 Jul 07 19:02:32.959473 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:32.959439 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd6pq" event={"ID":"015fa7a7-c259-40ff-9404-c9d00bcb1e17","Type":"ContainerStarted","Data":"cd9503f0a999258d815eacfe3225701dbbce46ad8ede51a49dd9126df5061503"} Jul 07 19:02:33.965426 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:33.965390 2559 generic.go:356] "Generic (PLEG): container finished" podID="015fa7a7-c259-40ff-9404-c9d00bcb1e17" containerID="78977b28b4882aef2e906a03cd4c52025b2f58cb586f8efbb6d8fdb7ac914209" exitCode=0 Jul 07 19:02:33.965914 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:33.965462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd6pq" event={"ID":"015fa7a7-c259-40ff-9404-c9d00bcb1e17","Type":"ContainerDied","Data":"78977b28b4882aef2e906a03cd4c52025b2f58cb586f8efbb6d8fdb7ac914209"} Jul 07 19:02:34.970759 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:34.970723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd6pq" event={"ID":"015fa7a7-c259-40ff-9404-c9d00bcb1e17","Type":"ContainerStarted","Data":"d6f80e5c15155ffdffb1b3f0dc5323741e1a93cc146d0a32f69f3ad98b65aa7e"} Jul 07 19:02:34.971300 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:34.970784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nd6pq" event={"ID":"015fa7a7-c259-40ff-9404-c9d00bcb1e17","Type":"ContainerStarted","Data":"a0d5c0803dca927407f80c066cf58938763fc85d4de200d37b83c2a4f2dc044c"} Jul 07 19:02:34.992899 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:34.992848 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nd6pq" podStartSLOduration=2.9719489770000003 podStartE2EDuration="3.99283212s" totalImagesPullingTime="955.651484ms" totalInitContainerRuntime="65.231659ms" isStatefulPod=true podCreationTimestamp="2026-07-07 19:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:34.991625136 +0000 UTC m=+80.013275141" watchObservedRunningTime="2026-07-07 19:02:34.99283212 +0000 UTC m=+80.014482123" Jul 07 19:02:37.512658 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.512545 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b7db6484c-c72xr"] Jul 07 19:02:37.514689 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.514672 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.517451 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.517426 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 07 19:02:37.517451 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.517443 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 07 19:02:37.518492 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.518474 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 07 19:02:37.518492 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.518480 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 07 19:02:37.518619 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.518484 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-t59jx\"" Jul 07 19:02:37.518744 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.518724 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 07 19:02:37.518796 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.518779 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 07 19:02:37.518847 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.518795 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 07 19:02:37.523971 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.523950 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 07 19:02:37.527494 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.527471 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7db6484c-c72xr"] Jul 07 19:02:37.668363 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.668324 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-oauth-config\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.668363 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.668364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-config\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.668585 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.668384 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-trusted-ca-bundle\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.668585 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.668404 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-oauth-serving-cert\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.668585 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.668422 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-service-ca\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.668585 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.668494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcmb\" (UniqueName: \"kubernetes.io/projected/f5efaad4-a147-4ed1-a355-a4e38596d43a-kube-api-access-cgcmb\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.668585 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.668524 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-serving-cert\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.768959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.768863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcmb\" (UniqueName: \"kubernetes.io/projected/f5efaad4-a147-4ed1-a355-a4e38596d43a-kube-api-access-cgcmb\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.768959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.768906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-serving-cert\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.768959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.768960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-oauth-config\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.769255 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.768979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-config\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.769255 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.768997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-trusted-ca-bundle\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.769255 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.769021 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-oauth-serving-cert\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.769255 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.769045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-service-ca\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.769902 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.769877 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-service-ca\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.770003 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.769905 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-config\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.770003 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.769913 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-oauth-serving-cert\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.770441 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.770417 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-trusted-ca-bundle\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.771617 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.771585 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-oauth-config\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.771773 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.771755 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-serving-cert\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.778330 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.778307 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcmb\" (UniqueName: \"kubernetes.io/projected/f5efaad4-a147-4ed1-a355-a4e38596d43a-kube-api-access-cgcmb\") pod \"console-7b7db6484c-c72xr\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.825340 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.825295 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:37.976585 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:37.976553 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b7db6484c-c72xr"] Jul 07 19:02:37.979515 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:02:37.979485 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5efaad4_a147_4ed1_a355_a4e38596d43a.slice/crio-79d6e3190f81de18cfa584580c7e52687ddaadb0881c611b5051719ccd70331f WatchSource:0}: Error finding container 79d6e3190f81de18cfa584580c7e52687ddaadb0881c611b5051719ccd70331f: Status 404 returned error can't find the container with id 79d6e3190f81de18cfa584580c7e52687ddaadb0881c611b5051719ccd70331f Jul 07 19:02:38.984901 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:38.984866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7db6484c-c72xr" event={"ID":"f5efaad4-a147-4ed1-a355-a4e38596d43a","Type":"ContainerStarted","Data":"79d6e3190f81de18cfa584580c7e52687ddaadb0881c611b5051719ccd70331f"} Jul 07 19:02:40.993777 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:40.993732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7db6484c-c72xr" event={"ID":"f5efaad4-a147-4ed1-a355-a4e38596d43a","Type":"ContainerStarted","Data":"9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0"} Jul 07 19:02:41.014483 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:41.014430 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b7db6484c-c72xr" podStartSLOduration=1.1701405089999999 podStartE2EDuration="4.014413613s" totalImagesPullingTime="2.844273104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:02:41.01351716 +0000 UTC m=+86.035167164" watchObservedRunningTime="2026-07-07 19:02:41.014413613 +0000 UTC m=+86.036063617" Jul 07 19:02:41.413082 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:41.413049 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7db6484c-c72xr"] Jul 07 19:02:47.825447 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:47.825388 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:02:59.949949 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:02:59.949921 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h8sbv" Jul 07 19:03:08.019789 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.019718 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7b7db6484c-c72xr" podUID="f5efaad4-a147-4ed1-a355-a4e38596d43a" containerName="console" containerID="cri-o://9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0" gracePeriod=15 Jul 07 19:03:08.258295 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.258266 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:03:08.297495 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297409 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-trusted-ca-bundle\") pod \"f5efaad4-a147-4ed1-a355-a4e38596d43a\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " Jul 07 19:03:08.297495 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297451 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-oauth-config\") pod \"f5efaad4-a147-4ed1-a355-a4e38596d43a\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " Jul 07 19:03:08.297495 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297480 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-serving-cert\") pod \"f5efaad4-a147-4ed1-a355-a4e38596d43a\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " Jul 07 19:03:08.297766 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297504 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-service-ca\") pod \"f5efaad4-a147-4ed1-a355-a4e38596d43a\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " Jul 07 19:03:08.297766 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297538 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-config\") pod \"f5efaad4-a147-4ed1-a355-a4e38596d43a\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " Jul 07 19:03:08.297766 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297575 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-oauth-serving-cert\") pod \"f5efaad4-a147-4ed1-a355-a4e38596d43a\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " Jul 07 19:03:08.297766 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297602 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgcmb\" (UniqueName: \"kubernetes.io/projected/f5efaad4-a147-4ed1-a355-a4e38596d43a-kube-api-access-cgcmb\") pod \"f5efaad4-a147-4ed1-a355-a4e38596d43a\" (UID: \"f5efaad4-a147-4ed1-a355-a4e38596d43a\") " Jul 07 19:03:08.297987 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.297961 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f5efaad4-a147-4ed1-a355-a4e38596d43a" (UID: "f5efaad4-a147-4ed1-a355-a4e38596d43a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:08.298142 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.298115 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-service-ca" (OuterVolumeSpecName: "service-ca") pod "f5efaad4-a147-4ed1-a355-a4e38596d43a" (UID: "f5efaad4-a147-4ed1-a355-a4e38596d43a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:08.298249 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.298144 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-config" (OuterVolumeSpecName: "console-config") pod "f5efaad4-a147-4ed1-a355-a4e38596d43a" (UID: "f5efaad4-a147-4ed1-a355-a4e38596d43a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:08.298249 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.298156 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f5efaad4-a147-4ed1-a355-a4e38596d43a" (UID: "f5efaad4-a147-4ed1-a355-a4e38596d43a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 19:03:08.303286 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.301300 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5efaad4-a147-4ed1-a355-a4e38596d43a-kube-api-access-cgcmb" (OuterVolumeSpecName: "kube-api-access-cgcmb") pod "f5efaad4-a147-4ed1-a355-a4e38596d43a" (UID: "f5efaad4-a147-4ed1-a355-a4e38596d43a"). InnerVolumeSpecName "kube-api-access-cgcmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:03:08.303658 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.303626 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f5efaad4-a147-4ed1-a355-a4e38596d43a" (UID: "f5efaad4-a147-4ed1-a355-a4e38596d43a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:08.303768 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.303650 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f5efaad4-a147-4ed1-a355-a4e38596d43a" (UID: "f5efaad4-a147-4ed1-a355-a4e38596d43a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:03:08.399078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.399040 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-trusted-ca-bundle\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:03:08.399078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.399069 2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-oauth-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:03:08.399078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.399080 2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-serving-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:03:08.399078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.399089 2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-service-ca\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:03:08.399358 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.399098 2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-console-config\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:03:08.399358 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.399127 2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f5efaad4-a147-4ed1-a355-a4e38596d43a-oauth-serving-cert\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:03:08.399358 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:08.399136 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcmb\" (UniqueName: \"kubernetes.io/projected/f5efaad4-a147-4ed1-a355-a4e38596d43a-kube-api-access-cgcmb\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:03:09.088527 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.088491 2559 generic.go:356] "Generic (PLEG): container finished" podID="f5efaad4-a147-4ed1-a355-a4e38596d43a" containerID="9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0" exitCode=0 Jul 07 19:03:09.088975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.088548 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b7db6484c-c72xr" Jul 07 19:03:09.088975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.088576 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7db6484c-c72xr" event={"ID":"f5efaad4-a147-4ed1-a355-a4e38596d43a","Type":"ContainerDied","Data":"9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0"} Jul 07 19:03:09.088975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.088625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b7db6484c-c72xr" event={"ID":"f5efaad4-a147-4ed1-a355-a4e38596d43a","Type":"ContainerDied","Data":"79d6e3190f81de18cfa584580c7e52687ddaadb0881c611b5051719ccd70331f"} Jul 07 19:03:09.088975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.088647 2559 scope.go:117] "RemoveContainer" containerID="9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0" Jul 07 19:03:09.097705 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.097686 2559 scope.go:117] "RemoveContainer" containerID="9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0" Jul 07 19:03:09.098000 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:03:09.097982 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0\": container with ID starting with 9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0 not found: ID does not exist" containerID="9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0" Jul 07 19:03:09.098068 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.098012 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0"} err="failed to get container status \"9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0\": rpc error: code = NotFound desc = could not find container \"9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0\": container with ID starting with 9464faf959059edf14b456f33f36c93738961918061d444c8ae2fb8907259db0 not found: ID does not exist" Jul 07 19:03:09.108487 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.108454 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b7db6484c-c72xr"] Jul 07 19:03:09.114191 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.114149 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b7db6484c-c72xr"] Jul 07 19:03:09.595634 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:03:09.595599 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5efaad4-a147-4ed1-a355-a4e38596d43a" path="/var/lib/kubelet/pods/f5efaad4-a147-4ed1-a355-a4e38596d43a/volumes" Jul 07 19:05:23.679491 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.679435 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pwffw"] Jul 07 19:05:23.680053 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.679793 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5efaad4-a147-4ed1-a355-a4e38596d43a" containerName="console" Jul 07 19:05:23.680053 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.679814 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5efaad4-a147-4ed1-a355-a4e38596d43a" containerName="console" Jul 07 19:05:23.680053 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.679876 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5efaad4-a147-4ed1-a355-a4e38596d43a" containerName="console" Jul 07 19:05:23.682701 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.682678 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:23.685141 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.685113 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-khvl7\"" Jul 07 19:05:23.685141 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.685120 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 19:05:23.685922 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.685902 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 19:05:23.691928 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.691900 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pwffw"] Jul 07 19:05:23.727775 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.727731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88348e39-1be0-4339-9769-826933636cd3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pwffw\" (UID: \"88348e39-1be0-4339-9769-826933636cd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:23.727959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.727784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzgm\" (UniqueName: \"kubernetes.io/projected/88348e39-1be0-4339-9769-826933636cd3-kube-api-access-xdzgm\") pod \"cert-manager-cainjector-559cdd8c87-pwffw\" (UID: \"88348e39-1be0-4339-9769-826933636cd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:23.829155 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.829094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzgm\" (UniqueName: \"kubernetes.io/projected/88348e39-1be0-4339-9769-826933636cd3-kube-api-access-xdzgm\") pod \"cert-manager-cainjector-559cdd8c87-pwffw\" (UID: \"88348e39-1be0-4339-9769-826933636cd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:23.829155 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.829193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88348e39-1be0-4339-9769-826933636cd3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pwffw\" (UID: \"88348e39-1be0-4339-9769-826933636cd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:23.838489 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.838452 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88348e39-1be0-4339-9769-826933636cd3-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pwffw\" (UID: \"88348e39-1be0-4339-9769-826933636cd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:23.838654 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.838597 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzgm\" (UniqueName: \"kubernetes.io/projected/88348e39-1be0-4339-9769-826933636cd3-kube-api-access-xdzgm\") pod \"cert-manager-cainjector-559cdd8c87-pwffw\" (UID: \"88348e39-1be0-4339-9769-826933636cd3\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:23.991789 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:23.991689 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" Jul 07 19:05:24.138072 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:05:24.138034 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88348e39_1be0_4339_9769_826933636cd3.slice/crio-e8d93abd5b40329ac129645badd2a497cb92130d7d8d4f5c02a9c9a044b7c4f4 WatchSource:0}: Error finding container e8d93abd5b40329ac129645badd2a497cb92130d7d8d4f5c02a9c9a044b7c4f4: Status 404 returned error can't find the container with id e8d93abd5b40329ac129645badd2a497cb92130d7d8d4f5c02a9c9a044b7c4f4 Jul 07 19:05:24.138839 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:24.138812 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pwffw"] Jul 07 19:05:24.534234 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:24.534198 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" event={"ID":"88348e39-1be0-4339-9769-826933636cd3","Type":"ContainerStarted","Data":"e8d93abd5b40329ac129645badd2a497cb92130d7d8d4f5c02a9c9a044b7c4f4"} Jul 07 19:05:27.548124 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:27.548089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" event={"ID":"88348e39-1be0-4339-9769-826933636cd3","Type":"ContainerStarted","Data":"e65015fea317bc3d1222e404a2de7824853cd2ab3f7c8d169bc35cf0d9821fa3"} Jul 07 19:05:27.562890 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:27.562834 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pwffw" podStartSLOduration=1.797643025 podStartE2EDuration="4.562816732s" totalImagesPullingTime="2.765173707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:05:27.561988484 +0000 UTC m=+252.583638482" watchObservedRunningTime="2026-07-07 19:05:27.562816732 +0000 UTC m=+252.584466736" Jul 07 19:05:51.403494 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.403411 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w"] Jul 07 19:05:51.407719 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.407697 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.411259 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.411230 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 19:05:51.411393 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.411243 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 19:05:51.411393 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.411313 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 19:05:51.411553 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.411541 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sflhs\"" Jul 07 19:05:51.412196 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.412180 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 19:05:51.425747 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.425715 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w"] Jul 07 19:05:51.528087 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.528051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1cf01b44-67ab-400c-8d26-9da703350f0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.528087 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.528094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1cf01b44-67ab-400c-8d26-9da703350f0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.528331 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.528117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cf01b44-67ab-400c-8d26-9da703350f0d-tmp\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.528331 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.528154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45qs\" (UniqueName: \"kubernetes.io/projected/1cf01b44-67ab-400c-8d26-9da703350f0d-kube-api-access-j45qs\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.528331 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.528223 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cf01b44-67ab-400c-8d26-9da703350f0d-manifests\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.629049 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.629012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45qs\" (UniqueName: \"kubernetes.io/projected/1cf01b44-67ab-400c-8d26-9da703350f0d-kube-api-access-j45qs\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.629049 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.629048 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cf01b44-67ab-400c-8d26-9da703350f0d-manifests\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.629346 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.629092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1cf01b44-67ab-400c-8d26-9da703350f0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.629346 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.629111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1cf01b44-67ab-400c-8d26-9da703350f0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.629346 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.629129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cf01b44-67ab-400c-8d26-9da703350f0d-tmp\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.629549 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.629527 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1cf01b44-67ab-400c-8d26-9da703350f0d-manifests\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.631422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.631399 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1cf01b44-67ab-400c-8d26-9da703350f0d-tmp\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.631558 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.631536 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1cf01b44-67ab-400c-8d26-9da703350f0d-webhook-cert\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.631723 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.631662 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1cf01b44-67ab-400c-8d26-9da703350f0d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.639880 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.639841 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45qs\" (UniqueName: \"kubernetes.io/projected/1cf01b44-67ab-400c-8d26-9da703350f0d-kube-api-access-j45qs\") pod \"opendatahub-operator-controller-manager-59996cbd9d-4bz6w\" (UID: \"1cf01b44-67ab-400c-8d26-9da703350f0d\") " pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.717926 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.717824 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:51.864578 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:51.864548 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w"] Jul 07 19:05:51.867391 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:05:51.867361 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf01b44_67ab_400c_8d26_9da703350f0d.slice/crio-ec974c382d44e323e6430594d900a50e3bfec226bcbae58881b50754e53d1aad WatchSource:0}: Error finding container ec974c382d44e323e6430594d900a50e3bfec226bcbae58881b50754e53d1aad: Status 404 returned error can't find the container with id ec974c382d44e323e6430594d900a50e3bfec226bcbae58881b50754e53d1aad Jul 07 19:05:52.632365 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:52.632332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" event={"ID":"1cf01b44-67ab-400c-8d26-9da703350f0d","Type":"ContainerStarted","Data":"ec974c382d44e323e6430594d900a50e3bfec226bcbae58881b50754e53d1aad"} Jul 07 19:05:55.644804 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:55.644766 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" event={"ID":"1cf01b44-67ab-400c-8d26-9da703350f0d","Type":"ContainerStarted","Data":"72ca0b70907b80b2ab8aade7799d548b2702ef1674e2c47f6a9eae575517e25b"} Jul 07 19:05:56.649249 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:56.649207 2559 generic.go:356] "Generic (PLEG): container finished" podID="1cf01b44-67ab-400c-8d26-9da703350f0d" containerID="72ca0b70907b80b2ab8aade7799d548b2702ef1674e2c47f6a9eae575517e25b" exitCode=0 Jul 07 19:05:56.649634 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:56.649267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" event={"ID":"1cf01b44-67ab-400c-8d26-9da703350f0d","Type":"ContainerDied","Data":"72ca0b70907b80b2ab8aade7799d548b2702ef1674e2c47f6a9eae575517e25b"} Jul 07 19:05:57.654930 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:57.654887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" event={"ID":"1cf01b44-67ab-400c-8d26-9da703350f0d","Type":"ContainerStarted","Data":"03835f9329934d3a421b6fbafb6f86f135366d3ceca57b818359886eceaf385d"} Jul 07 19:05:57.655339 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:57.655034 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:05:57.678051 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:05:57.677991 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" podStartSLOduration=2.047683654 podStartE2EDuration="6.677971534s" totalImagesPullingTime="3.613368435s" totalInitContainerRuntime="1.016919445s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:05:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:05:57.676307648 +0000 UTC m=+282.697957653" watchObservedRunningTime="2026-07-07 19:05:57.677971534 +0000 UTC m=+282.699621538" Jul 07 19:06:08.660582 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:08.660550 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-59996cbd9d-4bz6w" Jul 07 19:06:11.110426 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.110388 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wbp6w"] Jul 07 19:06:11.112373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.112354 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.114797 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.114771 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 07 19:06:11.114927 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.114849 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gn5lg\"" Jul 07 19:06:11.124422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.124394 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wbp6w"] Jul 07 19:06:11.280786 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.280742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpmkk\" (UniqueName: \"kubernetes.io/projected/e672aeec-2a0a-45f0-b295-70412ac69d22-kube-api-access-cpmkk\") pod \"odh-model-controller-7998c67dc4-wbp6w\" (UID: \"e672aeec-2a0a-45f0-b295-70412ac69d22\") " pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.280786 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.280787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e672aeec-2a0a-45f0-b295-70412ac69d22-cert\") pod \"odh-model-controller-7998c67dc4-wbp6w\" (UID: \"e672aeec-2a0a-45f0-b295-70412ac69d22\") " pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.381471 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.381381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpmkk\" (UniqueName: \"kubernetes.io/projected/e672aeec-2a0a-45f0-b295-70412ac69d22-kube-api-access-cpmkk\") pod \"odh-model-controller-7998c67dc4-wbp6w\" (UID: \"e672aeec-2a0a-45f0-b295-70412ac69d22\") " pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.381471 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.381421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e672aeec-2a0a-45f0-b295-70412ac69d22-cert\") pod \"odh-model-controller-7998c67dc4-wbp6w\" (UID: \"e672aeec-2a0a-45f0-b295-70412ac69d22\") " pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.383907 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.383881 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e672aeec-2a0a-45f0-b295-70412ac69d22-cert\") pod \"odh-model-controller-7998c67dc4-wbp6w\" (UID: \"e672aeec-2a0a-45f0-b295-70412ac69d22\") " pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.392557 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.392514 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpmkk\" (UniqueName: \"kubernetes.io/projected/e672aeec-2a0a-45f0-b295-70412ac69d22-kube-api-access-cpmkk\") pod \"odh-model-controller-7998c67dc4-wbp6w\" (UID: \"e672aeec-2a0a-45f0-b295-70412ac69d22\") " pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.422387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.422350 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:11.557495 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.557460 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wbp6w"] Jul 07 19:06:11.560275 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:06:11.560245 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode672aeec_2a0a_45f0_b295_70412ac69d22.slice/crio-aacd6f68fc6509194e0ca42b38ddb6f8e2800cd462a857073e7a55e6812e315b WatchSource:0}: Error finding container aacd6f68fc6509194e0ca42b38ddb6f8e2800cd462a857073e7a55e6812e315b: Status 404 returned error can't find the container with id aacd6f68fc6509194e0ca42b38ddb6f8e2800cd462a857073e7a55e6812e315b Jul 07 19:06:11.695741 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.695644 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-955bd7587-mp5tm"] Jul 07 19:06:11.698373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.698346 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.700492 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.700467 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 19:06:11.700723 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.700701 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 19:06:11.700853 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.700809 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 19:06:11.701075 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.701057 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 19:06:11.701298 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.701284 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-jqwcr\"" Jul 07 19:06:11.705867 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.705835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" event={"ID":"e672aeec-2a0a-45f0-b295-70412ac69d22","Type":"ContainerStarted","Data":"aacd6f68fc6509194e0ca42b38ddb6f8e2800cd462a857073e7a55e6812e315b"} Jul 07 19:06:11.710607 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.710578 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-955bd7587-mp5tm"] Jul 07 19:06:11.785519 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.785470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458762dd-80a2-44a9-a6f0-fd1918ed4a70-tmp\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.785702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.785552 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxc9\" (UniqueName: \"kubernetes.io/projected/458762dd-80a2-44a9-a6f0-fd1918ed4a70-kube-api-access-rpxc9\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.785702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.785593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458762dd-80a2-44a9-a6f0-fd1918ed4a70-tls-certs\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.886560 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.886511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458762dd-80a2-44a9-a6f0-fd1918ed4a70-tmp\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.886560 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.886567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxc9\" (UniqueName: \"kubernetes.io/projected/458762dd-80a2-44a9-a6f0-fd1918ed4a70-kube-api-access-rpxc9\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.886816 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.886593 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458762dd-80a2-44a9-a6f0-fd1918ed4a70-tls-certs\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.888834 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.888814 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458762dd-80a2-44a9-a6f0-fd1918ed4a70-tmp\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.888951 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.888934 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/458762dd-80a2-44a9-a6f0-fd1918ed4a70-tls-certs\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:11.900454 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:11.900419 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxc9\" (UniqueName: \"kubernetes.io/projected/458762dd-80a2-44a9-a6f0-fd1918ed4a70-kube-api-access-rpxc9\") pod \"kube-auth-proxy-955bd7587-mp5tm\" (UID: \"458762dd-80a2-44a9-a6f0-fd1918ed4a70\") " pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:12.011141 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:12.011049 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" Jul 07 19:06:12.156949 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:12.156837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-955bd7587-mp5tm"] Jul 07 19:06:12.159438 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:06:12.159408 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458762dd_80a2_44a9_a6f0_fd1918ed4a70.slice/crio-ae51f15240036e901fa558acb9131dbcd98aa0647052fb5b3b990404f9f35139 WatchSource:0}: Error finding container ae51f15240036e901fa558acb9131dbcd98aa0647052fb5b3b990404f9f35139: Status 404 returned error can't find the container with id ae51f15240036e901fa558acb9131dbcd98aa0647052fb5b3b990404f9f35139 Jul 07 19:06:12.710707 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:12.710666 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" event={"ID":"458762dd-80a2-44a9-a6f0-fd1918ed4a70","Type":"ContainerStarted","Data":"ae51f15240036e901fa558acb9131dbcd98aa0647052fb5b3b990404f9f35139"} Jul 07 19:06:15.491815 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.491792 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 19:06:15.497182 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.497147 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:06:15.497352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.497333 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:06:15.504441 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.504422 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 19:06:15.722760 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.722720 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" event={"ID":"e672aeec-2a0a-45f0-b295-70412ac69d22","Type":"ContainerStarted","Data":"cc9455a599a48863ec7a954768de2683eb69d69b46a6ca50d0e5bf43a45528f4"} Jul 07 19:06:15.724148 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.722832 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:15.724269 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.724200 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" event={"ID":"458762dd-80a2-44a9-a6f0-fd1918ed4a70","Type":"ContainerStarted","Data":"ca4871027e7e0f1dfd0e88241dfd74dbadb809716d143ff1a340b220780a6f4e"} Jul 07 19:06:15.739655 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.739550 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" podStartSLOduration=0.810308629 podStartE2EDuration="4.739533604s" totalImagesPullingTime="3.929224975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:15.738082806 +0000 UTC m=+300.759732821" watchObservedRunningTime="2026-07-07 19:06:15.739533604 +0000 UTC m=+300.761183606" Jul 07 19:06:15.755504 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:15.755443 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-955bd7587-mp5tm" podStartSLOduration=1.42783477 podStartE2EDuration="4.755422913s" totalImagesPullingTime="3.327588143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:15.753730871 +0000 UTC m=+300.775380875" watchObservedRunningTime="2026-07-07 19:06:15.755422913 +0000 UTC m=+300.777072920" Jul 07 19:06:17.935592 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:17.935555 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z"] Jul 07 19:06:17.937776 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:17.937758 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:17.940943 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:17.940917 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zhxvd\"" Jul 07 19:06:17.941126 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:17.941108 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 07 19:06:17.955895 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:17.955862 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z"] Jul 07 19:06:18.040959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.040920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041151 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.040967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041151 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.041004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041151 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.041024 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041151 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.041087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041151 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.041118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041364 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.041158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041364 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.041221 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n96hs\" (UniqueName: \"kubernetes.io/projected/261cf77c-43a3-4f06-9c0d-8ea9b666615e-kube-api-access-n96hs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.041364 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.041268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142095 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142095 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n96hs\" (UniqueName: \"kubernetes.io/projected/261cf77c-43a3-4f06-9c0d-8ea9b666615e-kube-api-access-n96hs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142630 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142688 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142638 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142688 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142661 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142824 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142798 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.142899 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.142881 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.143154 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.143132 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.144571 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.144535 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.144776 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.144757 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.156608 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.156568 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n96hs\" (UniqueName: \"kubernetes.io/projected/261cf77c-43a3-4f06-9c0d-8ea9b666615e-kube-api-access-n96hs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.168270 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.168240 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/261cf77c-43a3-4f06-9c0d-8ea9b666615e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf45k75z\" (UID: \"261cf77c-43a3-4f06-9c0d-8ea9b666615e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.248101 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.248011 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:18.403921 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.403887 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z"] Jul 07 19:06:18.407009 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:06:18.406982 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261cf77c_43a3_4f06_9c0d_8ea9b666615e.slice/crio-8680b1aa6c1ffba008620646ab57303c8c033d20861a583a6ce2fa6bf17b7802 WatchSource:0}: Error finding container 8680b1aa6c1ffba008620646ab57303c8c033d20861a583a6ce2fa6bf17b7802: Status 404 returned error can't find the container with id 8680b1aa6c1ffba008620646ab57303c8c033d20861a583a6ce2fa6bf17b7802 Jul 07 19:06:18.409314 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.409298 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:06:18.583631 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.583545 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx"] Jul 07 19:06:18.586680 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.586652 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.601430 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.601408 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 19:06:18.601632 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.601434 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 19:06:18.601632 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.601408 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2w9rd\"" Jul 07 19:06:18.601632 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.601408 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 19:06:18.601792 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.601729 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 19:06:18.604353 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.604330 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx"] Jul 07 19:06:18.605631 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.605611 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 19:06:18.735450 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.735416 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" event={"ID":"261cf77c-43a3-4f06-9c0d-8ea9b666615e","Type":"ContainerStarted","Data":"8680b1aa6c1ffba008620646ab57303c8c033d20861a583a6ce2fa6bf17b7802"} Jul 07 19:06:18.746853 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.746818 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-manager-config\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.746853 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.746859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-metrics-cert\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.747061 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.746887 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmxfn\" (UniqueName: \"kubernetes.io/projected/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-kube-api-access-gmxfn\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.747061 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.746931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-cert\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.847456 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.847406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-manager-config\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.847652 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.847496 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-metrics-cert\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.847652 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.847554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmxfn\" (UniqueName: \"kubernetes.io/projected/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-kube-api-access-gmxfn\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.847652 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.847590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-cert\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.848377 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.848278 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-manager-config\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.850465 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.850444 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-cert\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.850616 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.850597 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-metrics-cert\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.857220 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.857125 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmxfn\" (UniqueName: \"kubernetes.io/projected/209bd0c1-fade-461e-a4cf-ac5eeb85cac2-kube-api-access-gmxfn\") pod \"lws-controller-manager-6c8bfdd549-5nzsx\" (UID: \"209bd0c1-fade-461e-a4cf-ac5eeb85cac2\") " pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:18.896058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:18.896013 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:19.074307 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:19.074270 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx"] Jul 07 19:06:19.224590 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:06:19.224481 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209bd0c1_fade_461e_a4cf_ac5eeb85cac2.slice/crio-a22547fa01c27f0b7ecd1bb91c5cad497be6f168a13d3a9b4c634d88264e7ea6 WatchSource:0}: Error finding container a22547fa01c27f0b7ecd1bb91c5cad497be6f168a13d3a9b4c634d88264e7ea6: Status 404 returned error can't find the container with id a22547fa01c27f0b7ecd1bb91c5cad497be6f168a13d3a9b4c634d88264e7ea6 Jul 07 19:06:19.742143 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:19.742095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" event={"ID":"209bd0c1-fade-461e-a4cf-ac5eeb85cac2","Type":"ContainerStarted","Data":"a22547fa01c27f0b7ecd1bb91c5cad497be6f168a13d3a9b4c634d88264e7ea6"} Jul 07 19:06:22.030611 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.030559 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:06:22.031058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.030661 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:06:22.031058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.030698 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:06:22.754973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.754931 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" event={"ID":"261cf77c-43a3-4f06-9c0d-8ea9b666615e","Type":"ContainerStarted","Data":"551931489ff42206c97b8c9d3366df97396b36a702aea0a02f3cd2477b92045e"} Jul 07 19:06:22.756433 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.756405 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" event={"ID":"209bd0c1-fade-461e-a4cf-ac5eeb85cac2","Type":"ContainerStarted","Data":"15a4de7b83abfa59004344e7c04bc05a7dc8394e5142c57885faaa5502f94faa"} Jul 07 19:06:22.756566 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.756464 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:06:22.780634 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.780580 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" podStartSLOduration=2.159738484 podStartE2EDuration="5.780564541s" totalImagesPullingTime="3.620826057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:22.7788089 +0000 UTC m=+307.800458904" watchObservedRunningTime="2026-07-07 19:06:22.780564541 +0000 UTC m=+307.802214611" Jul 07 19:06:22.799008 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:22.798953 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" podStartSLOduration=1.984979788 podStartE2EDuration="4.798936948s" totalImagesPullingTime="2.81395716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:06:22.797846029 +0000 UTC m=+307.819496034" watchObservedRunningTime="2026-07-07 19:06:22.798936948 +0000 UTC m=+307.820587014" Jul 07 19:06:23.248765 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:23.248720 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:23.253291 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:23.253269 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:23.760232 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:23.760196 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:23.761062 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:23.761042 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf45k75z" Jul 07 19:06:26.729770 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:26.729741 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wbp6w" Jul 07 19:06:33.761917 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:06:33.761883 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6c8bfdd549-5nzsx" Jul 07 19:08:53.744112 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.744081 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c"] Jul 07 19:08:53.747399 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.747375 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.749976 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.749951 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4mxs4\"" Jul 07 19:08:53.756306 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756439 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756317 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756439 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756375 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756439 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756401 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756439 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3626d9-37a5-4954-bca8-671444c66047-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756648 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756481 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756648 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3626d9-37a5-4954-bca8-671444c66047-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756648 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756579 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf2sf\" (UniqueName: \"kubernetes.io/projected/7c3626d9-37a5-4954-bca8-671444c66047-kube-api-access-nf2sf\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.756648 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.756610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3626d9-37a5-4954-bca8-671444c66047-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.760296 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.760268 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c"] Jul 07 19:08:53.857090 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3626d9-37a5-4954-bca8-671444c66047-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857105 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3626d9-37a5-4954-bca8-671444c66047-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf2sf\" (UniqueName: \"kubernetes.io/projected/7c3626d9-37a5-4954-bca8-671444c66047-kube-api-access-nf2sf\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3626d9-37a5-4954-bca8-671444c66047-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857267 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857691 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857691 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857556 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857691 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857627 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857827 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857737 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857871 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857834 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c3626d9-37a5-4954-bca8-671444c66047-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.857938 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.857913 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.858149 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.858132 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.859729 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.859704 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c3626d9-37a5-4954-bca8-671444c66047-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.860028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.860008 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c3626d9-37a5-4954-bca8-671444c66047-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.865704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.865673 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c3626d9-37a5-4954-bca8-671444c66047-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:53.866294 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:53.866269 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf2sf\" (UniqueName: \"kubernetes.io/projected/7c3626d9-37a5-4954-bca8-671444c66047-kube-api-access-nf2sf\") pod \"maas-default-gateway-openshift-default-56654dcf7-62l8c\" (UID: \"7c3626d9-37a5-4954-bca8-671444c66047\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:54.060572 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.060476 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:54.218183 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.218142 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c"] Jul 07 19:08:54.221833 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:08:54.221804 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3626d9_37a5_4954_bca8_671444c66047.slice/crio-3474a8f1ac2d975a879ac76c925db6fdaf07318c4c89e090f3466fd1e46c2565 WatchSource:0}: Error finding container 3474a8f1ac2d975a879ac76c925db6fdaf07318c4c89e090f3466fd1e46c2565: Status 404 returned error can't find the container with id 3474a8f1ac2d975a879ac76c925db6fdaf07318c4c89e090f3466fd1e46c2565 Jul 07 19:08:54.223893 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.223864 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:08:54.223958 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.223934 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:08:54.223996 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.223967 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:08:54.316255 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.316147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" event={"ID":"7c3626d9-37a5-4954-bca8-671444c66047","Type":"ContainerStarted","Data":"38660a237c286904c4a4cdc18635373a264de5ae81de9a5e65dbdeab9b7ab8f1"} Jul 07 19:08:54.316255 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.316218 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" event={"ID":"7c3626d9-37a5-4954-bca8-671444c66047","Type":"ContainerStarted","Data":"3474a8f1ac2d975a879ac76c925db6fdaf07318c4c89e090f3466fd1e46c2565"} Jul 07 19:08:54.342146 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:54.342076 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" podStartSLOduration=1.342051746 podStartE2EDuration="1.342051746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:08:54.339449234 +0000 UTC m=+459.361099251" watchObservedRunningTime="2026-07-07 19:08:54.342051746 +0000 UTC m=+459.363701751" Jul 07 19:08:55.060997 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:55.060955 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:55.065901 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:55.065875 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:55.320804 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:55.320715 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:08:55.321823 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:08:55.321804 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:09:11.800160 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.800120 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-b5l6w"] Jul 07 19:09:11.803347 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.803325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b5l6w" Jul 07 19:09:11.806204 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.806176 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 19:09:11.807202 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.807181 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 19:09:11.807320 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.807190 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qxc8p\"" Jul 07 19:09:11.841555 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.841520 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b5l6w"] Jul 07 19:09:11.896756 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.896715 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxh8\" (UniqueName: \"kubernetes.io/projected/fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50-kube-api-access-5dxh8\") pod \"authorino-58cd955675-b5l6w\" (UID: \"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50\") " pod="kuadrant-system/authorino-58cd955675-b5l6w" Jul 07 19:09:11.997732 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:11.997692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxh8\" (UniqueName: \"kubernetes.io/projected/fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50-kube-api-access-5dxh8\") pod \"authorino-58cd955675-b5l6w\" (UID: \"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50\") " pod="kuadrant-system/authorino-58cd955675-b5l6w" Jul 07 19:09:12.013382 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:12.013349 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxh8\" (UniqueName: \"kubernetes.io/projected/fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50-kube-api-access-5dxh8\") pod \"authorino-58cd955675-b5l6w\" (UID: \"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50\") " pod="kuadrant-system/authorino-58cd955675-b5l6w" Jul 07 19:09:12.112951 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:12.112910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b5l6w" Jul 07 19:09:12.256244 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:12.256216 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-b5l6w"] Jul 07 19:09:12.258669 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:09:12.258642 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd2b96d1_b5a4_4f14_a4ae_c9b1abb4dd50.slice/crio-f87a9e34139bed2bc531bd72dcb7eb8b67819f34325a2ae8a79cb048ea60c3a5 WatchSource:0}: Error finding container f87a9e34139bed2bc531bd72dcb7eb8b67819f34325a2ae8a79cb048ea60c3a5: Status 404 returned error can't find the container with id f87a9e34139bed2bc531bd72dcb7eb8b67819f34325a2ae8a79cb048ea60c3a5 Jul 07 19:09:12.388577 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:12.388487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b5l6w" event={"ID":"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50","Type":"ContainerStarted","Data":"f87a9e34139bed2bc531bd72dcb7eb8b67819f34325a2ae8a79cb048ea60c3a5"} Jul 07 19:09:16.408257 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:16.408221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b5l6w" event={"ID":"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50","Type":"ContainerStarted","Data":"2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba"} Jul 07 19:09:16.455427 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:16.455375 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-b5l6w" podStartSLOduration=1.449007934 podStartE2EDuration="5.45535441s" totalImagesPullingTime="4.006346476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:09:16.451350206 +0000 UTC m=+481.473000192" watchObservedRunningTime="2026-07-07 19:09:16.45535441 +0000 UTC m=+481.477004413" Jul 07 19:09:16.899538 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:16.899505 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b5l6w"] Jul 07 19:09:18.416085 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:18.416042 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-b5l6w" podUID="fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50" containerName="authorino" containerID="cri-o://2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba" gracePeriod=30 Jul 07 19:09:18.665085 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:18.665051 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b5l6w" Jul 07 19:09:18.758562 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:18.758473 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dxh8\" (UniqueName: \"kubernetes.io/projected/fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50-kube-api-access-5dxh8\") pod \"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50\" (UID: \"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50\") " Jul 07 19:09:18.760673 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:18.760643 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50-kube-api-access-5dxh8" (OuterVolumeSpecName: "kube-api-access-5dxh8") pod "fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50" (UID: "fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50"). InnerVolumeSpecName "kube-api-access-5dxh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:09:18.859123 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:18.859091 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxh8\" (UniqueName: \"kubernetes.io/projected/fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50-kube-api-access-5dxh8\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:09:19.426457 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.426413 2559 generic.go:356] "Generic (PLEG): container finished" podID="fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50" containerID="2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba" exitCode=0 Jul 07 19:09:19.426934 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.426505 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-b5l6w" Jul 07 19:09:19.426934 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.426501 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b5l6w" event={"ID":"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50","Type":"ContainerDied","Data":"2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba"} Jul 07 19:09:19.426934 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.426577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-b5l6w" event={"ID":"fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50","Type":"ContainerDied","Data":"f87a9e34139bed2bc531bd72dcb7eb8b67819f34325a2ae8a79cb048ea60c3a5"} Jul 07 19:09:19.426934 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.426605 2559 scope.go:117] "RemoveContainer" containerID="2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba" Jul 07 19:09:19.434910 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.434888 2559 scope.go:117] "RemoveContainer" containerID="2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba" Jul 07 19:09:19.435241 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:09:19.435215 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba\": container with ID starting with 2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba not found: ID does not exist" containerID="2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba" Jul 07 19:09:19.435352 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.435248 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba"} err="failed to get container status \"2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba\": rpc error: code = NotFound desc = could not find container \"2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba\": container with ID starting with 2cd6f4f1086982d5ade49d1ab2486b435542dec4002126ed0a34a9844cc842ba not found: ID does not exist" Jul 07 19:09:19.457400 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.457365 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-b5l6w"] Jul 07 19:09:19.463573 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.463544 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-b5l6w"] Jul 07 19:09:19.595523 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:09:19.595489 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50" path="/var/lib/kubelet/pods/fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50/volumes" Jul 07 19:10:36.732271 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:36.732153 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c3626d9-37a5-4954-bca8-671444c66047" containerID="38660a237c286904c4a4cdc18635373a264de5ae81de9a5e65dbdeab9b7ab8f1" exitCode=0 Jul 07 19:10:36.732271 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:36.732225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" event={"ID":"7c3626d9-37a5-4954-bca8-671444c66047","Type":"ContainerDied","Data":"38660a237c286904c4a4cdc18635373a264de5ae81de9a5e65dbdeab9b7ab8f1"} Jul 07 19:10:36.732681 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:36.732618 2559 scope.go:117] "RemoveContainer" containerID="38660a237c286904c4a4cdc18635373a264de5ae81de9a5e65dbdeab9b7ab8f1" Jul 07 19:10:36.733586 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:36.733562 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:10:36.733649 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:36.733621 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:10:36.733683 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:36.733652 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 07 19:10:37.738017 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:37.737978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" event={"ID":"7c3626d9-37a5-4954-bca8-671444c66047","Type":"ContainerStarted","Data":"1620ddbd6e34521ea1c900429d439e65b59f7ad91a378b2faaf56f9ae3fc2c44"} Jul 07 19:10:38.061238 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:38.061138 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:10:38.062385 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:38.062344 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-62l8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Jul 07 19:10:38.062519 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:38.062408 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" podUID="7c3626d9-37a5-4954-bca8-671444c66047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Jul 07 19:10:39.061791 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:39.061754 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-62l8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Jul 07 19:10:39.062219 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:39.061808 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" podUID="7c3626d9-37a5-4954-bca8-671444c66047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Jul 07 19:10:40.061270 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:40.061234 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-62l8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Jul 07 19:10:40.061454 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:40.061289 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" podUID="7c3626d9-37a5-4954-bca8-671444c66047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Jul 07 19:10:40.321002 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:40.320899 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:10:41.061079 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:41.061037 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-62l8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" start-of-body= Jul 07 19:10:41.061289 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:41.061108 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" podUID="7c3626d9-37a5-4954-bca8-671444c66047" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.20:15021/healthz/ready\": dial tcp 10.134.0.20:15021: connect: connection refused" Jul 07 19:10:42.065159 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:42.065128 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:10:42.066246 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:42.066223 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62l8c" Jul 07 19:10:48.016810 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.016779 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc"] Jul 07 19:10:48.017308 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.017058 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50" containerName="authorino" Jul 07 19:10:48.017308 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.017068 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50" containerName="authorino" Jul 07 19:10:48.017308 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.017117 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2b96d1-b5a4-4f14-a4ae-c9b1abb4dd50" containerName="authorino" Jul 07 19:10:48.018910 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.018894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.022740 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.022715 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8m2wx\"" Jul 07 19:10:48.022890 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.022746 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 07 19:10:48.022890 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.022751 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 19:10:48.022890 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.022715 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 19:10:48.036349 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.036317 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc"] Jul 07 19:10:48.181762 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.181714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.181762 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.181761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.182009 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.181795 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkcjg\" (UniqueName: \"kubernetes.io/projected/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-kube-api-access-gkcjg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.182009 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.181881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.182009 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.181925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.182009 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.181946 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.282567 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.282470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.282567 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.282513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.282787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.282642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkcjg\" (UniqueName: \"kubernetes.io/projected/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-kube-api-access-gkcjg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.282787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.282692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.282787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.282723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.282787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.282746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.282972 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.282876 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.283023 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.283002 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.283148 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.283114 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.284769 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.284751 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.285096 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.285080 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.292885 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.292859 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkcjg\" (UniqueName: \"kubernetes.io/projected/0a321d62-4f1f-4e34-9953-5b3e40ad5b52-kube-api-access-gkcjg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc\" (UID: \"0a321d62-4f1f-4e34-9953-5b3e40ad5b52\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.329528 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.329488 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:48.483297 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.483264 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc"] Jul 07 19:10:48.486382 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:10:48.486352 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a321d62_4f1f_4e34_9953_5b3e40ad5b52.slice/crio-8f19ec82dd8ae275c4e6403f465393b48d8ba1d525744d6fdc7c02d9f07906b0 WatchSource:0}: Error finding container 8f19ec82dd8ae275c4e6403f465393b48d8ba1d525744d6fdc7c02d9f07906b0: Status 404 returned error can't find the container with id 8f19ec82dd8ae275c4e6403f465393b48d8ba1d525744d6fdc7c02d9f07906b0 Jul 07 19:10:48.781967 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:48.781932 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" event={"ID":"0a321d62-4f1f-4e34-9953-5b3e40ad5b52","Type":"ContainerStarted","Data":"8f19ec82dd8ae275c4e6403f465393b48d8ba1d525744d6fdc7c02d9f07906b0"} Jul 07 19:10:49.788469 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:49.788433 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" event={"ID":"0a321d62-4f1f-4e34-9953-5b3e40ad5b52","Type":"ContainerStarted","Data":"452a043d4cab53ad90a726e363335c43b8b7c0236e1dbb75d529e225f9446480"} Jul 07 19:10:49.809606 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:49.809526 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" podStartSLOduration=1.605327653 podStartE2EDuration="2.809510467s" totalImagesPullingTime="1.204182814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:10:49.808950571 +0000 UTC m=+574.830600576" watchObservedRunningTime="2026-07-07 19:10:49.809510467 +0000 UTC m=+574.831160470" Jul 07 19:10:58.330649 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:58.330600 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:58.331125 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:58.330687 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:58.343346 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:58.343312 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:10:58.835518 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:10:58.835485 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc" Jul 07 19:11:15.521746 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:15.521715 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:11:15.522266 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:15.522223 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:11:19.902022 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.901987 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67"] Jul 07 19:11:19.905421 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.905395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:19.908523 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.908493 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 19:11:19.914684 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.914654 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67"] Jul 07 19:11:19.940279 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.940237 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:19.940279 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.940284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:19.940532 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.940307 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/14992779-c28a-4285-8b65-904b8efaf99d-kube-api-access-h5zrt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:19.940532 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.940405 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:19.940532 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.940437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14992779-c28a-4285-8b65-904b8efaf99d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:19.940532 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:19.940506 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.041686 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.041642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.041686 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.041693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.041948 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.041719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/14992779-c28a-4285-8b65-904b8efaf99d-kube-api-access-h5zrt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.041948 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.041762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.041948 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.041783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14992779-c28a-4285-8b65-904b8efaf99d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.041948 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.041834 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.042207 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.042111 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.042207 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.042141 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.042293 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.042227 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.044128 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.044108 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14992779-c28a-4285-8b65-904b8efaf99d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.044240 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.044222 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14992779-c28a-4285-8b65-904b8efaf99d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.051658 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.051624 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zrt\" (UniqueName: \"kubernetes.io/projected/14992779-c28a-4285-8b65-904b8efaf99d-kube-api-access-h5zrt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67\" (UID: \"14992779-c28a-4285-8b65-904b8efaf99d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.217261 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.217153 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:20.349309 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.349140 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67"] Jul 07 19:11:20.351948 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:11:20.351922 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14992779_c28a_4285_8b65_904b8efaf99d.slice/crio-5d318429a8e951b774efda30e037e303c7b33a714df1c69c6fcef3defb0b2b7f WatchSource:0}: Error finding container 5d318429a8e951b774efda30e037e303c7b33a714df1c69c6fcef3defb0b2b7f: Status 404 returned error can't find the container with id 5d318429a8e951b774efda30e037e303c7b33a714df1c69c6fcef3defb0b2b7f Jul 07 19:11:20.354088 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.354071 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 19:11:20.913190 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.913076 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" event={"ID":"14992779-c28a-4285-8b65-904b8efaf99d","Type":"ContainerStarted","Data":"977981f6df0d794d3361d24dd61519231a7f7bb11a9a3fc3dac7d2f4b0da47bb"} Jul 07 19:11:20.913190 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.913129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" event={"ID":"14992779-c28a-4285-8b65-904b8efaf99d","Type":"ContainerStarted","Data":"5d318429a8e951b774efda30e037e303c7b33a714df1c69c6fcef3defb0b2b7f"} Jul 07 19:11:20.932363 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:20.932291 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" podStartSLOduration=1.694766652 podStartE2EDuration="1.932273518s" totalImagesPullingTime="237.506866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:20.931457603 +0000 UTC m=+605.953107608" watchObservedRunningTime="2026-07-07 19:11:20.932273518 +0000 UTC m=+605.953923521" Jul 07 19:11:30.217559 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:30.217498 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:30.218110 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:30.217589 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:30.230828 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:30.230796 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:30.969017 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:30.968984 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67" Jul 07 19:11:31.786898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.786862 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r"] Jul 07 19:11:31.792274 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.792249 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:31.795363 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.795338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 07 19:11:31.803323 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.803283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r"] Jul 07 19:11:31.937343 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.937303 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1928cd31-a85d-4235-937c-66944ccf2fed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:31.937526 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.937359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:31.937526 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.937406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:31.937526 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.937451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:31.937526 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.937494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz75w\" (UniqueName: \"kubernetes.io/projected/1928cd31-a85d-4235-937c-66944ccf2fed-kube-api-access-bz75w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:31.937659 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:31.937533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.038601 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.038491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.038601 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.038558 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1928cd31-a85d-4235-937c-66944ccf2fed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.038839 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.038633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.038839 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.038661 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.038839 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.038714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.038839 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.038752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz75w\" (UniqueName: \"kubernetes.io/projected/1928cd31-a85d-4235-937c-66944ccf2fed-kube-api-access-bz75w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.039121 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.039011 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.039214 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.039116 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.039214 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.039151 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.041140 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.041110 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1928cd31-a85d-4235-937c-66944ccf2fed-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.041281 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.041266 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1928cd31-a85d-4235-937c-66944ccf2fed-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.049999 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.049967 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz75w\" (UniqueName: \"kubernetes.io/projected/1928cd31-a85d-4235-937c-66944ccf2fed-kube-api-access-bz75w\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5854r\" (UID: \"1928cd31-a85d-4235-937c-66944ccf2fed\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.103427 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.103390 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:32.241818 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.241792 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r"] Jul 07 19:11:32.244209 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:11:32.244181 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1928cd31_a85d_4235_937c_66944ccf2fed.slice/crio-08463d7cd07b8fc5aef6977b299db07d8361df7ee7b6214f5cc85d6cb4df4c03 WatchSource:0}: Error finding container 08463d7cd07b8fc5aef6977b299db07d8361df7ee7b6214f5cc85d6cb4df4c03: Status 404 returned error can't find the container with id 08463d7cd07b8fc5aef6977b299db07d8361df7ee7b6214f5cc85d6cb4df4c03 Jul 07 19:11:32.967966 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.967927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" event={"ID":"1928cd31-a85d-4235-937c-66944ccf2fed","Type":"ContainerStarted","Data":"b50e0ba7e9ffc1e678d4255772e68b30082a8965ad431dd46a6d073b856739f9"} Jul 07 19:11:32.968424 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.967981 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" event={"ID":"1928cd31-a85d-4235-937c-66944ccf2fed","Type":"ContainerStarted","Data":"08463d7cd07b8fc5aef6977b299db07d8361df7ee7b6214f5cc85d6cb4df4c03"} Jul 07 19:11:32.986997 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:32.986939 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" podStartSLOduration=1.7730020770000001 podStartE2EDuration="1.986919834s" totalImagesPullingTime="213.917757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:32.985664485 +0000 UTC m=+618.007314489" watchObservedRunningTime="2026-07-07 19:11:32.986919834 +0000 UTC m=+618.008569837" Jul 07 19:11:42.103975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:42.103874 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:42.103975 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:42.103942 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:42.116533 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:42.116504 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:43.022591 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:43.022558 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5854r" Jul 07 19:11:48.195402 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.195365 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh"] Jul 07 19:11:48.198475 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.198451 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.203050 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.203027 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 19:11:48.213286 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.213249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh"] Jul 07 19:11:48.284486 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.284447 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.284486 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.284486 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cc5a57-070e-4909-941d-f5f37c0525f1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.284712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.284512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.284712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.284581 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.284712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.284631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8x8j\" (UniqueName: \"kubernetes.io/projected/88cc5a57-070e-4909-941d-f5f37c0525f1-kube-api-access-r8x8j\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.284712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.284677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385099 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385099 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cc5a57-070e-4909-941d-f5f37c0525f1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385396 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385396 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385209 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385396 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8x8j\" (UniqueName: \"kubernetes.io/projected/88cc5a57-070e-4909-941d-f5f37c0525f1-kube-api-access-r8x8j\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385396 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385303 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385633 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385609 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385707 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385654 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.385763 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.385710 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.387528 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.387505 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88cc5a57-070e-4909-941d-f5f37c0525f1-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.387822 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.387801 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88cc5a57-070e-4909-941d-f5f37c0525f1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.396818 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.396787 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8x8j\" (UniqueName: \"kubernetes.io/projected/88cc5a57-070e-4909-941d-f5f37c0525f1-kube-api-access-r8x8j\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh\" (UID: \"88cc5a57-070e-4909-941d-f5f37c0525f1\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.508533 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.508433 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:48.646350 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:48.646314 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh"] Jul 07 19:11:48.648557 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:11:48.648525 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88cc5a57_070e_4909_941d_f5f37c0525f1.slice/crio-25ba52e05f353993aac75429209579211179cb5f685e81c96047b787d6a84e36 WatchSource:0}: Error finding container 25ba52e05f353993aac75429209579211179cb5f685e81c96047b787d6a84e36: Status 404 returned error can't find the container with id 25ba52e05f353993aac75429209579211179cb5f685e81c96047b787d6a84e36 Jul 07 19:11:49.039734 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:49.039695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" event={"ID":"88cc5a57-070e-4909-941d-f5f37c0525f1","Type":"ContainerStarted","Data":"3da60c8b51395dfc5321210e32ce678d98df867702bc17fcbad34801c80f32ec"} Jul 07 19:11:49.039734 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:49.039739 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" event={"ID":"88cc5a57-070e-4909-941d-f5f37c0525f1","Type":"ContainerStarted","Data":"25ba52e05f353993aac75429209579211179cb5f685e81c96047b787d6a84e36"} Jul 07 19:11:49.061241 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:49.061180 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" podStartSLOduration=0.868601257 podStartE2EDuration="1.061149859s" totalImagesPullingTime="192.548602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:11:49.059820344 +0000 UTC m=+634.081470351" watchObservedRunningTime="2026-07-07 19:11:49.061149859 +0000 UTC m=+634.082799863" Jul 07 19:11:58.508854 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:58.508818 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:58.508854 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:58.508869 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:58.521610 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:58.521579 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:11:59.093492 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:11:59.093456 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh" Jul 07 19:16:15.545050 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:16:15.545017 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:16:15.546458 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:16:15.546436 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:21:15.567966 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:21:15.567930 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:21:15.570881 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:21:15.570861 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:26:15.595068 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:26:15.595036 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:26:15.598686 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:26:15.598664 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:29:26.096767 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.096724 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7"] Jul 07 19:29:26.100152 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.100128 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.102702 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.102678 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-13b76f9e\"/\"kube-root-ca.crt\"" Jul 07 19:29:26.102895 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.102865 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-13b76f9e\"/\"default-dockercfg-q7mc7\"" Jul 07 19:29:26.104245 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.104223 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-13b76f9e\"/\"openshift-service-ca.crt\"" Jul 07 19:29:26.104365 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.104249 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-13b76f9e\"/\"e2e-lifecycle-model-13b76f9e-kserve-self-signed-certs\"" Jul 07 19:29:26.115427 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.115390 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7"] Jul 07 19:29:26.139235 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.139199 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwptw\" (UniqueName: \"kubernetes.io/projected/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-kube-api-access-bwptw\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.139422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.139255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-home\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.139422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.139326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tmp-dir\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.139422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.139372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-dshm\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.139422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.139390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tls-certs\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.139557 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.139479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-model-cache\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240241 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-home\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240241 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tmp-dir\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240497 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-dshm\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240497 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tls-certs\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240497 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-model-cache\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240497 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwptw\" (UniqueName: \"kubernetes.io/projected/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-kube-api-access-bwptw\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240608 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-home\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240766 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240710 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tmp-dir\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.240813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.240765 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-model-cache\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.242460 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.242443 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-dshm\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.242826 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.242804 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tls-certs\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.249076 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.249047 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwptw\" (UniqueName: \"kubernetes.io/projected/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-kube-api-access-bwptw\") pod \"e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.411690 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.411588 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:26.550798 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:26.550771 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7"] Jul 07 19:29:26.552646 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:29:26.552613 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d7b7829_6a35_4dbe_9ae2_73da887cfac9.slice/crio-9dc9dc2aaeff80f6db14a08f819574d2802a30f4c6e5985ead69674452d315ea WatchSource:0}: Error finding container 9dc9dc2aaeff80f6db14a08f819574d2802a30f4c6e5985ead69674452d315ea: Status 404 returned error can't find the container with id 9dc9dc2aaeff80f6db14a08f819574d2802a30f4c6e5985ead69674452d315ea Jul 07 19:29:27.496090 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:27.496052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" event={"ID":"5d7b7829-6a35-4dbe-9ae2-73da887cfac9","Type":"ContainerStarted","Data":"17cbe144c1628ecce4595da6e1cd1f2746435dbb90db70b1a8624236c51fc46b"} Jul 07 19:29:27.496494 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:27.496101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" event={"ID":"5d7b7829-6a35-4dbe-9ae2-73da887cfac9","Type":"ContainerStarted","Data":"9dc9dc2aaeff80f6db14a08f819574d2802a30f4c6e5985ead69674452d315ea"} Jul 07 19:29:27.514047 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:27.513997 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" podStartSLOduration=1.513978105 podStartE2EDuration="1.513978105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:29:27.513215367 +0000 UTC m=+1692.534865370" watchObservedRunningTime="2026-07-07 19:29:27.513978105 +0000 UTC m=+1692.535628126" Jul 07 19:29:36.411993 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:36.411883 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:36.411993 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:36.411960 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:36.424591 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:36.424561 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:29:36.544488 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:29:36.544458 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:30:07.571943 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:07.571902 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7"] Jul 07 19:30:22.592546 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.592503 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" podUID="5d7b7829-6a35-4dbe-9ae2-73da887cfac9" containerName="main" containerID="cri-o://17cbe144c1628ecce4595da6e1cd1f2746435dbb90db70b1a8624236c51fc46b" gracePeriod=45 Jul 07 19:30:22.731543 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.731509 2559 generic.go:356] "Generic (PLEG): container finished" podID="5d7b7829-6a35-4dbe-9ae2-73da887cfac9" containerID="17cbe144c1628ecce4595da6e1cd1f2746435dbb90db70b1a8624236c51fc46b" exitCode=0 Jul 07 19:30:22.731708 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.731568 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" event={"ID":"5d7b7829-6a35-4dbe-9ae2-73da887cfac9","Type":"ContainerDied","Data":"17cbe144c1628ecce4595da6e1cd1f2746435dbb90db70b1a8624236c51fc46b"} Jul 07 19:30:22.848058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.847984 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:30:22.933419 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933375 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tls-certs\") pod \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " Jul 07 19:30:22.933606 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933442 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-dshm\") pod \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " Jul 07 19:30:22.933606 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933469 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tmp-dir\") pod \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " Jul 07 19:30:22.933606 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933567 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-model-cache\") pod \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " Jul 07 19:30:22.933727 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933626 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwptw\" (UniqueName: \"kubernetes.io/projected/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-kube-api-access-bwptw\") pod \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " Jul 07 19:30:22.933727 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933668 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-home\") pod \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\" (UID: \"5d7b7829-6a35-4dbe-9ae2-73da887cfac9\") " Jul 07 19:30:22.933809 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933718 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d7b7829-6a35-4dbe-9ae2-73da887cfac9" (UID: "5d7b7829-6a35-4dbe-9ae2-73da887cfac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:22.933851 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933800 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d7b7829-6a35-4dbe-9ae2-73da887cfac9" (UID: "5d7b7829-6a35-4dbe-9ae2-73da887cfac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:22.933951 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.933931 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-home" (OuterVolumeSpecName: "home") pod "5d7b7829-6a35-4dbe-9ae2-73da887cfac9" (UID: "5d7b7829-6a35-4dbe-9ae2-73da887cfac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:22.934034 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.934009 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:30:22.934034 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.934036 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:30:22.935871 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.935839 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-kube-api-access-bwptw" (OuterVolumeSpecName: "kube-api-access-bwptw") pod "5d7b7829-6a35-4dbe-9ae2-73da887cfac9" (UID: "5d7b7829-6a35-4dbe-9ae2-73da887cfac9"). InnerVolumeSpecName "kube-api-access-bwptw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:30:22.936000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.935893 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d7b7829-6a35-4dbe-9ae2-73da887cfac9" (UID: "5d7b7829-6a35-4dbe-9ae2-73da887cfac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:30:22.936000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:22.935898 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-dshm" (OuterVolumeSpecName: "dshm") pod "5d7b7829-6a35-4dbe-9ae2-73da887cfac9" (UID: "5d7b7829-6a35-4dbe-9ae2-73da887cfac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:30:23.034701 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.034665 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwptw\" (UniqueName: \"kubernetes.io/projected/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-kube-api-access-bwptw\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:30:23.034701 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.034695 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:30:23.034701 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.034705 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:30:23.034701 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.034713 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d7b7829-6a35-4dbe-9ae2-73da887cfac9-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:30:23.737661 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.737630 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" Jul 07 19:30:23.737661 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.737653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7" event={"ID":"5d7b7829-6a35-4dbe-9ae2-73da887cfac9","Type":"ContainerDied","Data":"9dc9dc2aaeff80f6db14a08f819574d2802a30f4c6e5985ead69674452d315ea"} Jul 07 19:30:23.738386 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.737707 2559 scope.go:117] "RemoveContainer" containerID="17cbe144c1628ecce4595da6e1cd1f2746435dbb90db70b1a8624236c51fc46b" Jul 07 19:30:23.754278 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.754246 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7"] Jul 07 19:30:23.758460 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:23.758431 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-13b76f9e/e2e-lifecycle-model-13b76f9e-kserve-7f4cbc8d95-xtfj7"] Jul 07 19:30:25.596728 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:30:25.596687 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7b7829-6a35-4dbe-9ae2-73da887cfac9" path="/var/lib/kubelet/pods/5d7b7829-6a35-4dbe-9ae2-73da887cfac9/volumes" Jul 07 19:31:15.622021 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:31:15.621986 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:31:15.626080 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:31:15.626054 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:32:23.119051 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.119008 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj"] Jul 07 19:32:23.119704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.119488 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7b7829-6a35-4dbe-9ae2-73da887cfac9" containerName="main" Jul 07 19:32:23.119704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.119508 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7b7829-6a35-4dbe-9ae2-73da887cfac9" containerName="main" Jul 07 19:32:23.119704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.119590 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7b7829-6a35-4dbe-9ae2-73da887cfac9" containerName="main" Jul 07 19:32:23.122503 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.122480 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.128003 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.127968 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a43967\"/\"openshift-service-ca.crt\"" Jul 07 19:32:23.128003 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.127995 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a43967\"/\"kube-root-ca.crt\"" Jul 07 19:32:23.128245 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.127967 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a43967\"/\"default-dockercfg-xdvst\"" Jul 07 19:32:23.128245 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.127967 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a43967\"/\"test-model-a43967-kserve-self-signed-certs\"" Jul 07 19:32:23.135059 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.135030 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj"] Jul 07 19:32:23.145830 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.145793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad8853-371e-4eb1-bed8-281629cf733d-tls-certs\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.146006 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.145854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-home\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.146006 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.145943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkhb\" (UniqueName: \"kubernetes.io/projected/d6ad8853-371e-4eb1-bed8-281629cf733d-kube-api-access-jpkhb\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.146126 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.146026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-dshm\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.146126 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.146061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-tmp-dir\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.146126 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.146095 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-model-cache\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.246512 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.246476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-dshm\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.246512 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.246515 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-tmp-dir\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.246768 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.246533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-model-cache\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.246768 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.246560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad8853-371e-4eb1-bed8-281629cf733d-tls-certs\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.246768 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.246606 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-home\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.246768 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.246684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkhb\" (UniqueName: \"kubernetes.io/projected/d6ad8853-371e-4eb1-bed8-281629cf733d-kube-api-access-jpkhb\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.247008 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.246985 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-tmp-dir\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.247061 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.247025 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-model-cache\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.247061 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.247049 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-home\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.248830 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.248805 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-dshm\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.249260 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.249238 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad8853-371e-4eb1-bed8-281629cf733d-tls-certs\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.258266 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.258239 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkhb\" (UniqueName: \"kubernetes.io/projected/d6ad8853-371e-4eb1-bed8-281629cf733d-kube-api-access-jpkhb\") pod \"test-model-a43967-kserve-66689968b9-t6cjj\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.431929 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.431834 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:23.597021 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:23.596993 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj"] Jul 07 19:32:23.598233 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:32:23.598208 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ad8853_371e_4eb1_bed8_281629cf733d.slice/crio-021d26207c0460efbc5e16d5c4d03a1143d644c7f6944bd8ce36c64ff40d00ec WatchSource:0}: Error finding container 021d26207c0460efbc5e16d5c4d03a1143d644c7f6944bd8ce36c64ff40d00ec: Status 404 returned error can't find the container with id 021d26207c0460efbc5e16d5c4d03a1143d644c7f6944bd8ce36c64ff40d00ec Jul 07 19:32:24.258586 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:24.258551 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" event={"ID":"d6ad8853-371e-4eb1-bed8-281629cf733d","Type":"ContainerStarted","Data":"5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a"} Jul 07 19:32:24.258970 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:24.258597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" event={"ID":"d6ad8853-371e-4eb1-bed8-281629cf733d","Type":"ContainerStarted","Data":"021d26207c0460efbc5e16d5c4d03a1143d644c7f6944bd8ce36c64ff40d00ec"} Jul 07 19:32:24.281456 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:24.281398 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" podStartSLOduration=1.281380454 podStartE2EDuration="1.281380454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:32:24.280189253 +0000 UTC m=+1869.301839259" watchObservedRunningTime="2026-07-07 19:32:24.281380454 +0000 UTC m=+1869.303030458" Jul 07 19:32:33.432546 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:33.432500 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:33.432959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:33.432566 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:33.445245 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:33.445206 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:34.318402 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:34.318370 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:32:41.003671 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.003581 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5"] Jul 07 19:32:41.007373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.007345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.011331 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.011283 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a4aa66\"/\"openshift-service-ca.crt\"" Jul 07 19:32:41.011331 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.011317 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a4aa66\"/\"default-dockercfg-j9lg5\"" Jul 07 19:32:41.011540 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.011299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a4aa66\"/\"test-model-a4aa66-kserve-self-signed-certs\"" Jul 07 19:32:41.011540 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.011297 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a4aa66\"/\"kube-root-ca.crt\"" Jul 07 19:32:41.023115 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.023080 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5"] Jul 07 19:32:41.092252 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.092211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-home\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.092252 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.092250 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tmp-dir\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.092487 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.092313 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwsg\" (UniqueName: \"kubernetes.io/projected/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-kube-api-access-qvwsg\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.092487 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.092391 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tls-certs\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.092487 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.092413 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-dshm\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.092487 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.092434 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-model-cache\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.193664 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.193603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-home\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.193664 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.193664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tmp-dir\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.193929 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.193708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwsg\" (UniqueName: \"kubernetes.io/projected/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-kube-api-access-qvwsg\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.193929 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.193760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tls-certs\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.193929 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.193800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-dshm\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.194079 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.193976 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-model-cache\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.194145 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.194117 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-home\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.194232 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.194130 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tmp-dir\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.194340 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.194321 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-model-cache\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.196142 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.196118 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-dshm\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.196449 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.196431 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tls-certs\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.205139 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.205110 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwsg\" (UniqueName: \"kubernetes.io/projected/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-kube-api-access-qvwsg\") pod \"test-model-a4aa66-kserve-777f4c967-h2nt5\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.317559 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.317453 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:41.489131 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:41.489095 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5"] Jul 07 19:32:41.492332 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:32:41.492303 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4e04a1c_55bc_4d37_b2f2_d8bfcbfe0939.slice/crio-1d13ca8078963e0dc2c8e823410d932041ba2abd83f075972a3533a9306113d5 WatchSource:0}: Error finding container 1d13ca8078963e0dc2c8e823410d932041ba2abd83f075972a3533a9306113d5: Status 404 returned error can't find the container with id 1d13ca8078963e0dc2c8e823410d932041ba2abd83f075972a3533a9306113d5 Jul 07 19:32:42.345382 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:42.345341 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" event={"ID":"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939","Type":"ContainerStarted","Data":"ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92"} Jul 07 19:32:42.345382 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:42.345389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" event={"ID":"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939","Type":"ContainerStarted","Data":"1d13ca8078963e0dc2c8e823410d932041ba2abd83f075972a3533a9306113d5"} Jul 07 19:32:42.369257 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:42.369202 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" podStartSLOduration=2.369182206 podStartE2EDuration="2.369182206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:32:42.367826717 +0000 UTC m=+1887.389476723" watchObservedRunningTime="2026-07-07 19:32:42.369182206 +0000 UTC m=+1887.390832202" Jul 07 19:32:51.318690 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:51.318637 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:51.318690 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:51.318695 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:51.331273 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:51.331230 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:32:51.403302 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:32:51.403271 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:33:02.635155 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:02.635119 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj"] Jul 07 19:33:03.604732 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:03.604697 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5"] Jul 07 19:33:17.655987 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.655940 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" podUID="d6ad8853-371e-4eb1-bed8-281629cf733d" containerName="main" containerID="cri-o://5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a" gracePeriod=45 Jul 07 19:33:17.895919 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.895892 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:33:17.935581 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935490 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-model-cache\") pod \"d6ad8853-371e-4eb1-bed8-281629cf733d\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " Jul 07 19:33:17.935581 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935569 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-tmp-dir\") pod \"d6ad8853-371e-4eb1-bed8-281629cf733d\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " Jul 07 19:33:17.935773 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935603 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-dshm\") pod \"d6ad8853-371e-4eb1-bed8-281629cf733d\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " Jul 07 19:33:17.935773 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935649 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad8853-371e-4eb1-bed8-281629cf733d-tls-certs\") pod \"d6ad8853-371e-4eb1-bed8-281629cf733d\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " Jul 07 19:33:17.935773 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935706 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-home\") pod \"d6ad8853-371e-4eb1-bed8-281629cf733d\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " Jul 07 19:33:17.935773 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935747 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpkhb\" (UniqueName: \"kubernetes.io/projected/d6ad8853-371e-4eb1-bed8-281629cf733d-kube-api-access-jpkhb\") pod \"d6ad8853-371e-4eb1-bed8-281629cf733d\" (UID: \"d6ad8853-371e-4eb1-bed8-281629cf733d\") " Jul 07 19:33:17.935968 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935774 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6ad8853-371e-4eb1-bed8-281629cf733d" (UID: "d6ad8853-371e-4eb1-bed8-281629cf733d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:17.936025 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935976 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:17.936025 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935979 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-home" (OuterVolumeSpecName: "home") pod "d6ad8853-371e-4eb1-bed8-281629cf733d" (UID: "d6ad8853-371e-4eb1-bed8-281629cf733d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:17.936025 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.935990 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6ad8853-371e-4eb1-bed8-281629cf733d" (UID: "d6ad8853-371e-4eb1-bed8-281629cf733d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:17.937899 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.937869 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-dshm" (OuterVolumeSpecName: "dshm") pod "d6ad8853-371e-4eb1-bed8-281629cf733d" (UID: "d6ad8853-371e-4eb1-bed8-281629cf733d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:17.938022 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.937935 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ad8853-371e-4eb1-bed8-281629cf733d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6ad8853-371e-4eb1-bed8-281629cf733d" (UID: "d6ad8853-371e-4eb1-bed8-281629cf733d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:33:17.938022 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:17.937967 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ad8853-371e-4eb1-bed8-281629cf733d-kube-api-access-jpkhb" (OuterVolumeSpecName: "kube-api-access-jpkhb") pod "d6ad8853-371e-4eb1-bed8-281629cf733d" (UID: "d6ad8853-371e-4eb1-bed8-281629cf733d"). InnerVolumeSpecName "kube-api-access-jpkhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:33:18.037058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.037019 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ad8853-371e-4eb1-bed8-281629cf733d-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:18.037058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.037048 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:18.037058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.037061 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpkhb\" (UniqueName: \"kubernetes.io/projected/d6ad8853-371e-4eb1-bed8-281629cf733d-kube-api-access-jpkhb\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:18.037315 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.037077 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:18.037315 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.037088 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ad8853-371e-4eb1-bed8-281629cf733d-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:18.511925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.511885 2559 generic.go:356] "Generic (PLEG): container finished" podID="d6ad8853-371e-4eb1-bed8-281629cf733d" containerID="5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a" exitCode=0 Jul 07 19:33:18.512133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.511930 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" Jul 07 19:33:18.512133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.511975 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" event={"ID":"d6ad8853-371e-4eb1-bed8-281629cf733d","Type":"ContainerDied","Data":"5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a"} Jul 07 19:33:18.512133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.512023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj" event={"ID":"d6ad8853-371e-4eb1-bed8-281629cf733d","Type":"ContainerDied","Data":"021d26207c0460efbc5e16d5c4d03a1143d644c7f6944bd8ce36c64ff40d00ec"} Jul 07 19:33:18.512133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.512044 2559 scope.go:117] "RemoveContainer" containerID="5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a" Jul 07 19:33:18.521506 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.521478 2559 scope.go:117] "RemoveContainer" containerID="5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a" Jul 07 19:33:18.521798 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:33:18.521774 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a\": container with ID starting with 5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a not found: ID does not exist" containerID="5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a" Jul 07 19:33:18.521854 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.521807 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a"} err="failed to get container status \"5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a\": rpc error: code = NotFound desc = could not find container \"5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a\": container with ID starting with 5f501f7e0a7bb5233b241d44c9aaabc7ccd332f7a7e46ad99b7722d069e7c13a not found: ID does not exist" Jul 07 19:33:18.541640 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.541606 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj"] Jul 07 19:33:18.544556 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.544530 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a43967/test-model-a43967-kserve-66689968b9-t6cjj"] Jul 07 19:33:18.624751 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.624705 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" podUID="b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" containerName="main" containerID="cri-o://ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92" gracePeriod=45 Jul 07 19:33:18.874906 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.874880 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:33:18.945681 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.945638 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tls-certs\") pod \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " Jul 07 19:33:18.945879 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.945711 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tmp-dir\") pod \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " Jul 07 19:33:18.945879 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.945811 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvwsg\" (UniqueName: \"kubernetes.io/projected/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-kube-api-access-qvwsg\") pod \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " Jul 07 19:33:18.945879 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.945846 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-dshm\") pod \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " Jul 07 19:33:18.946035 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.945904 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-model-cache\") pod \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " Jul 07 19:33:18.946035 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.945921 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-home\") pod \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\" (UID: \"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939\") " Jul 07 19:33:18.946035 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.945985 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" (UID: "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:18.946205 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.946162 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-home" (OuterVolumeSpecName: "home") pod "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" (UID: "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:18.946264 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.946206 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" (UID: "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:18.946264 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.946241 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:18.948105 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.948078 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" (UID: "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:33:18.948234 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.948080 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-dshm" (OuterVolumeSpecName: "dshm") pod "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" (UID: "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:33:18.948234 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:18.948120 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-kube-api-access-qvwsg" (OuterVolumeSpecName: "kube-api-access-qvwsg") pod "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" (UID: "b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939"). InnerVolumeSpecName "kube-api-access-qvwsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:33:19.046837 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.046729 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:19.046837 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.046776 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:19.046837 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.046785 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:19.046837 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.046794 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvwsg\" (UniqueName: \"kubernetes.io/projected/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-kube-api-access-qvwsg\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:19.046837 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.046803 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:33:19.518379 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.518341 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" containerID="ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92" exitCode=0 Jul 07 19:33:19.518572 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.518388 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" Jul 07 19:33:19.518572 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.518422 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" event={"ID":"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939","Type":"ContainerDied","Data":"ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92"} Jul 07 19:33:19.518572 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.518459 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5" event={"ID":"b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939","Type":"ContainerDied","Data":"1d13ca8078963e0dc2c8e823410d932041ba2abd83f075972a3533a9306113d5"} Jul 07 19:33:19.518572 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.518477 2559 scope.go:117] "RemoveContainer" containerID="ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92" Jul 07 19:33:19.527605 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.527582 2559 scope.go:117] "RemoveContainer" containerID="ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92" Jul 07 19:33:19.527932 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:33:19.527912 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92\": container with ID starting with ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92 not found: ID does not exist" containerID="ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92" Jul 07 19:33:19.527995 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.527946 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92"} err="failed to get container status \"ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92\": rpc error: code = NotFound desc = could not find container \"ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92\": container with ID starting with ae0247e7039e6d1ab963fe761f3b87cd263db0bd8451e7268c8cf8a42721db92 not found: ID does not exist" Jul 07 19:33:19.547327 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.547287 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5"] Jul 07 19:33:19.550440 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.550410 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a4aa66/test-model-a4aa66-kserve-777f4c967-h2nt5"] Jul 07 19:33:19.597467 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.597433 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" path="/var/lib/kubelet/pods/b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939/volumes" Jul 07 19:33:19.597861 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:33:19.597847 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ad8853-371e-4eb1-bed8-281629cf733d" path="/var/lib/kubelet/pods/d6ad8853-371e-4eb1-bed8-281629cf733d/volumes" Jul 07 19:34:52.613373 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.613333 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk"] Jul 07 19:34:52.613991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.613768 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ad8853-371e-4eb1-bed8-281629cf733d" containerName="main" Jul 07 19:34:52.613991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.613787 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ad8853-371e-4eb1-bed8-281629cf733d" containerName="main" Jul 07 19:34:52.613991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.613803 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" containerName="main" Jul 07 19:34:52.613991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.613811 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" containerName="main" Jul 07 19:34:52.613991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.613913 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e04a1c-55bc-4d37-b2f2-d8bfcbfe0939" containerName="main" Jul 07 19:34:52.613991 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.613926 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ad8853-371e-4eb1-bed8-281629cf733d" containerName="main" Jul 07 19:34:52.616876 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.616851 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.619637 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.619605 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c576d2\"/\"kube-root-ca.crt\"" Jul 07 19:34:52.620642 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.620618 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c576d2\"/\"openshift-service-ca.crt\"" Jul 07 19:34:52.620765 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.620639 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c576d2\"/\"default-dockercfg-ln6jn\"" Jul 07 19:34:52.620765 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.620618 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c576d2\"/\"auth-test-model-c576d2-kserve-self-signed-certs\"" Jul 07 19:34:52.631153 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.631121 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk"] Jul 07 19:34:52.774959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.774910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-model-cache\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.775211 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.774976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-dshm\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.775211 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.775080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-home\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.775211 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.775119 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tmp-dir\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.775367 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.775218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrbc4\" (UniqueName: \"kubernetes.io/projected/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-kube-api-access-rrbc4\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.775367 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.775260 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tls-certs\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.876607 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.876517 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-model-cache\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.876607 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.876575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-dshm\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.876820 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.876617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-home\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.876820 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.876633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tmp-dir\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.876820 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.876657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrbc4\" (UniqueName: \"kubernetes.io/projected/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-kube-api-access-rrbc4\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.876820 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.876688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tls-certs\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.877065 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.877041 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-model-cache\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.877065 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.877056 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-home\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.877191 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.877121 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tmp-dir\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.879424 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.879392 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-dshm\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.879567 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.879444 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tls-certs\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.885643 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.885617 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrbc4\" (UniqueName: \"kubernetes.io/projected/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-kube-api-access-rrbc4\") pod \"auth-test-model-c576d2-kserve-d64dbd76c-x54jk\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:52.926962 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:52.926921 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:34:53.071198 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:53.071114 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk"] Jul 07 19:34:53.073493 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:34:53.073454 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e52d1d8_5270_473b_8477_d1f1eda9fc7d.slice/crio-13d0488b3084edee16af01b5d3f5ecc6953a4df79d1a5d60b2aaabdf848acb77 WatchSource:0}: Error finding container 13d0488b3084edee16af01b5d3f5ecc6953a4df79d1a5d60b2aaabdf848acb77: Status 404 returned error can't find the container with id 13d0488b3084edee16af01b5d3f5ecc6953a4df79d1a5d60b2aaabdf848acb77 Jul 07 19:34:53.934966 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:53.934927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" event={"ID":"0e52d1d8-5270-473b-8477-d1f1eda9fc7d","Type":"ContainerStarted","Data":"f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d"} Jul 07 19:34:53.934966 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:53.934968 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" event={"ID":"0e52d1d8-5270-473b-8477-d1f1eda9fc7d","Type":"ContainerStarted","Data":"13d0488b3084edee16af01b5d3f5ecc6953a4df79d1a5d60b2aaabdf848acb77"} Jul 07 19:34:53.957981 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:34:53.957926 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" podStartSLOduration=1.9579083750000001 podStartE2EDuration="1.957908375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:34:53.956443171 +0000 UTC m=+2018.978093177" watchObservedRunningTime="2026-07-07 19:34:53.957908375 +0000 UTC m=+2018.979558379" Jul 07 19:35:02.927196 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:02.927120 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:35:02.927631 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:02.927211 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:35:02.939719 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:02.939691 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:35:02.987450 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:02.987415 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:35:13.500069 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.500024 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k"] Jul 07 19:35:13.503139 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.503123 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.505946 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.505920 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"openshift-service-ca.crt\"" Jul 07 19:35:13.506890 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.506868 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"auth-test-model-2620f1-kserve-self-signed-certs\"" Jul 07 19:35:13.507000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.506896 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"kube-root-ca.crt\"" Jul 07 19:35:13.507000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.506873 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"default-dockercfg-4dhwh\"" Jul 07 19:35:13.513236 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.513214 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k"] Jul 07 19:35:13.550473 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.550431 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-tmp-dir\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.550473 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.550472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p522\" (UniqueName: \"kubernetes.io/projected/53f5529d-e435-47a4-96c6-0029442df130-kube-api-access-7p522\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.550704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.550505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f5529d-e435-47a4-96c6-0029442df130-tls-certs\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.550704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.550618 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-home\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.550704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.550672 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-dshm\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.550814 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.550724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-model-cache\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651196 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651146 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-tmp-dir\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651196 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p522\" (UniqueName: \"kubernetes.io/projected/53f5529d-e435-47a4-96c6-0029442df130-kube-api-access-7p522\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651550 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651222 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f5529d-e435-47a4-96c6-0029442df130-tls-certs\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651550 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651267 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-home\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651550 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-dshm\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651550 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-model-cache\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651761 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651697 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-home\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651761 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651751 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-model-cache\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.651910 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.651880 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-tmp-dir\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.653808 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.653781 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-dshm\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.654534 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.654510 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f5529d-e435-47a4-96c6-0029442df130-tls-certs\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.661201 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.661150 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p522\" (UniqueName: \"kubernetes.io/projected/53f5529d-e435-47a4-96c6-0029442df130-kube-api-access-7p522\") pod \"auth-test-model-2620f1-kserve-59bd7d669f-qdz4k\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.812568 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.812466 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:13.955274 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:13.955243 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k"] Jul 07 19:35:13.958114 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:35:13.958083 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53f5529d_e435_47a4_96c6_0029442df130.slice/crio-a3cf214114089a3f0c2fe079a85c618e78984b12861e64f95b179563b994f0bb WatchSource:0}: Error finding container a3cf214114089a3f0c2fe079a85c618e78984b12861e64f95b179563b994f0bb: Status 404 returned error can't find the container with id a3cf214114089a3f0c2fe079a85c618e78984b12861e64f95b179563b994f0bb Jul 07 19:35:14.026209 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:14.026152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" event={"ID":"53f5529d-e435-47a4-96c6-0029442df130","Type":"ContainerStarted","Data":"88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6"} Jul 07 19:35:14.026336 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:14.026222 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" event={"ID":"53f5529d-e435-47a4-96c6-0029442df130","Type":"ContainerStarted","Data":"a3cf214114089a3f0c2fe079a85c618e78984b12861e64f95b179563b994f0bb"} Jul 07 19:35:14.044133 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:14.044076 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" podStartSLOduration=1.044056658 podStartE2EDuration="1.044056658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:35:14.043047496 +0000 UTC m=+2039.064697503" watchObservedRunningTime="2026-07-07 19:35:14.044056658 +0000 UTC m=+2039.065706661" Jul 07 19:35:23.813380 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:23.813338 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:23.813892 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:23.813398 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:23.826008 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:23.825976 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:35:24.079862 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:35:24.079771 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:36:15.647301 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:36:15.647154 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:36:15.652293 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:36:15.652270 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:37:04.497561 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:04.497522 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk"] Jul 07 19:37:05.485980 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:05.485938 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k"] Jul 07 19:37:11.203457 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.203379 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2"] Jul 07 19:37:11.206784 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.206764 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.209365 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.209343 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c576d2\"/\"sub-test-model-c576d2-kserve-self-signed-certs\"" Jul 07 19:37:11.218190 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.218143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2"] Jul 07 19:37:11.316777 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.316738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-home\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.316777 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.316779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-tmp-dir\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.317044 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.316865 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjgq6\" (UniqueName: \"kubernetes.io/projected/cdef644b-62e4-4383-9268-066bb8f0abf2-kube-api-access-xjgq6\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.317044 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.316909 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-dshm\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.317044 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.316933 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdef644b-62e4-4383-9268-066bb8f0abf2-tls-certs\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.317044 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.317010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-model-cache\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418202 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-home\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418404 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-tmp-dir\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418404 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjgq6\" (UniqueName: \"kubernetes.io/projected/cdef644b-62e4-4383-9268-066bb8f0abf2-kube-api-access-xjgq6\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418404 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-dshm\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418404 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418324 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdef644b-62e4-4383-9268-066bb8f0abf2-tls-certs\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418620 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418415 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-model-cache\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418680 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418613 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-home\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418680 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418668 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-tmp-dir\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.418806 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.418785 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-model-cache\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.420596 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.420577 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-dshm\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.421008 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.420985 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdef644b-62e4-4383-9268-066bb8f0abf2-tls-certs\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.428340 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.428300 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjgq6\" (UniqueName: \"kubernetes.io/projected/cdef644b-62e4-4383-9268-066bb8f0abf2-kube-api-access-xjgq6\") pod \"sub-test-model-c576d2-kserve-766d87645-vfpl2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.516157 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.516059 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:11.658207 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:11.658140 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2"] Jul 07 19:37:11.662885 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:37:11.662848 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdef644b_62e4_4383_9268_066bb8f0abf2.slice/crio-d0003f802f2172373b111cfa53b48e899b4d56f26cd3527e6e642b6d7463acbc WatchSource:0}: Error finding container d0003f802f2172373b111cfa53b48e899b4d56f26cd3527e6e642b6d7463acbc: Status 404 returned error can't find the container with id d0003f802f2172373b111cfa53b48e899b4d56f26cd3527e6e642b6d7463acbc Jul 07 19:37:12.557023 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:12.556976 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" event={"ID":"cdef644b-62e4-4383-9268-066bb8f0abf2","Type":"ContainerStarted","Data":"d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6"} Jul 07 19:37:12.557558 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:12.557036 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" event={"ID":"cdef644b-62e4-4383-9268-066bb8f0abf2","Type":"ContainerStarted","Data":"d0003f802f2172373b111cfa53b48e899b4d56f26cd3527e6e642b6d7463acbc"} Jul 07 19:37:12.574914 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:12.574856 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" podStartSLOduration=1.5748397029999999 podStartE2EDuration="1.574839703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:37:12.574180581 +0000 UTC m=+2157.595830581" watchObservedRunningTime="2026-07-07 19:37:12.574839703 +0000 UTC m=+2157.596489739" Jul 07 19:37:19.518018 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.517971 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" podUID="0e52d1d8-5270-473b-8477-d1f1eda9fc7d" containerName="main" containerID="cri-o://f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d" gracePeriod=45 Jul 07 19:37:19.768977 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.768904 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:37:19.793410 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793373 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-dshm\") pod \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " Jul 07 19:37:19.793582 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793454 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-home\") pod \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " Jul 07 19:37:19.793582 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793481 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-model-cache\") pod \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " Jul 07 19:37:19.793582 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793512 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tmp-dir\") pod \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " Jul 07 19:37:19.793582 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793545 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tls-certs\") pod \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " Jul 07 19:37:19.793777 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793595 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrbc4\" (UniqueName: \"kubernetes.io/projected/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-kube-api-access-rrbc4\") pod \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\" (UID: \"0e52d1d8-5270-473b-8477-d1f1eda9fc7d\") " Jul 07 19:37:19.793833 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793809 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-home" (OuterVolumeSpecName: "home") pod "0e52d1d8-5270-473b-8477-d1f1eda9fc7d" (UID: "0e52d1d8-5270-473b-8477-d1f1eda9fc7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:19.793833 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793824 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e52d1d8-5270-473b-8477-d1f1eda9fc7d" (UID: "0e52d1d8-5270-473b-8477-d1f1eda9fc7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:19.793942 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.793851 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e52d1d8-5270-473b-8477-d1f1eda9fc7d" (UID: "0e52d1d8-5270-473b-8477-d1f1eda9fc7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:19.794242 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.794216 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:19.794242 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.794242 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:19.794434 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.794256 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:19.795920 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.795892 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-dshm" (OuterVolumeSpecName: "dshm") pod "0e52d1d8-5270-473b-8477-d1f1eda9fc7d" (UID: "0e52d1d8-5270-473b-8477-d1f1eda9fc7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:19.796054 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.795941 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-kube-api-access-rrbc4" (OuterVolumeSpecName: "kube-api-access-rrbc4") pod "0e52d1d8-5270-473b-8477-d1f1eda9fc7d" (UID: "0e52d1d8-5270-473b-8477-d1f1eda9fc7d"). InnerVolumeSpecName "kube-api-access-rrbc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:37:19.796054 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.795971 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e52d1d8-5270-473b-8477-d1f1eda9fc7d" (UID: "0e52d1d8-5270-473b-8477-d1f1eda9fc7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:37:19.895428 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.895377 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:19.895428 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.895416 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrbc4\" (UniqueName: \"kubernetes.io/projected/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-kube-api-access-rrbc4\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:19.895428 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:19.895428 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e52d1d8-5270-473b-8477-d1f1eda9fc7d-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:20.507078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.507034 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" podUID="53f5529d-e435-47a4-96c6-0029442df130" containerName="main" containerID="cri-o://88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6" gracePeriod=45 Jul 07 19:37:20.598772 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.598730 2559 generic.go:356] "Generic (PLEG): container finished" podID="0e52d1d8-5270-473b-8477-d1f1eda9fc7d" containerID="f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d" exitCode=0 Jul 07 19:37:20.599222 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.598792 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" Jul 07 19:37:20.599222 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.598816 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" event={"ID":"0e52d1d8-5270-473b-8477-d1f1eda9fc7d","Type":"ContainerDied","Data":"f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d"} Jul 07 19:37:20.599222 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.598862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk" event={"ID":"0e52d1d8-5270-473b-8477-d1f1eda9fc7d","Type":"ContainerDied","Data":"13d0488b3084edee16af01b5d3f5ecc6953a4df79d1a5d60b2aaabdf848acb77"} Jul 07 19:37:20.599222 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.598880 2559 scope.go:117] "RemoveContainer" containerID="f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d" Jul 07 19:37:20.618632 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.618600 2559 scope.go:117] "RemoveContainer" containerID="f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d" Jul 07 19:37:20.618996 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:37:20.618958 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d\": container with ID starting with f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d not found: ID does not exist" containerID="f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d" Jul 07 19:37:20.619069 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.619011 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d"} err="failed to get container status \"f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d\": rpc error: code = NotFound desc = could not find container \"f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d\": container with ID starting with f998663270af8c770484c6029654783af3eb5be26d900d64155ccaebe1b42e7d not found: ID does not exist" Jul 07 19:37:20.620073 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.620050 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk"] Jul 07 19:37:20.623861 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.623834 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/auth-test-model-c576d2-kserve-d64dbd76c-x54jk"] Jul 07 19:37:20.768111 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.768039 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:37:20.803227 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f5529d-e435-47a4-96c6-0029442df130-tls-certs\") pod \"53f5529d-e435-47a4-96c6-0029442df130\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " Jul 07 19:37:20.803405 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803260 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-model-cache\") pod \"53f5529d-e435-47a4-96c6-0029442df130\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " Jul 07 19:37:20.803405 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803319 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p522\" (UniqueName: \"kubernetes.io/projected/53f5529d-e435-47a4-96c6-0029442df130-kube-api-access-7p522\") pod \"53f5529d-e435-47a4-96c6-0029442df130\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " Jul 07 19:37:20.803405 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803387 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-home\") pod \"53f5529d-e435-47a4-96c6-0029442df130\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " Jul 07 19:37:20.803593 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803438 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-dshm\") pod \"53f5529d-e435-47a4-96c6-0029442df130\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " Jul 07 19:37:20.803593 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803481 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-tmp-dir\") pod \"53f5529d-e435-47a4-96c6-0029442df130\" (UID: \"53f5529d-e435-47a4-96c6-0029442df130\") " Jul 07 19:37:20.803687 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803611 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-model-cache" (OuterVolumeSpecName: "model-cache") pod "53f5529d-e435-47a4-96c6-0029442df130" (UID: "53f5529d-e435-47a4-96c6-0029442df130"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:20.803687 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803671 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-home" (OuterVolumeSpecName: "home") pod "53f5529d-e435-47a4-96c6-0029442df130" (UID: "53f5529d-e435-47a4-96c6-0029442df130"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:20.803791 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803748 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53f5529d-e435-47a4-96c6-0029442df130" (UID: "53f5529d-e435-47a4-96c6-0029442df130"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:20.803851 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803810 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:20.803851 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803845 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:20.803944 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.803857 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:20.805763 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.805733 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-dshm" (OuterVolumeSpecName: "dshm") pod "53f5529d-e435-47a4-96c6-0029442df130" (UID: "53f5529d-e435-47a4-96c6-0029442df130"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:37:20.805763 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.805756 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f5529d-e435-47a4-96c6-0029442df130-kube-api-access-7p522" (OuterVolumeSpecName: "kube-api-access-7p522") pod "53f5529d-e435-47a4-96c6-0029442df130" (UID: "53f5529d-e435-47a4-96c6-0029442df130"). InnerVolumeSpecName "kube-api-access-7p522". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:37:20.805915 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.805786 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f5529d-e435-47a4-96c6-0029442df130-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53f5529d-e435-47a4-96c6-0029442df130" (UID: "53f5529d-e435-47a4-96c6-0029442df130"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:37:20.904888 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.904847 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53f5529d-e435-47a4-96c6-0029442df130-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:20.904888 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.904880 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p522\" (UniqueName: \"kubernetes.io/projected/53f5529d-e435-47a4-96c6-0029442df130-kube-api-access-7p522\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:20.904888 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:20.904891 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53f5529d-e435-47a4-96c6-0029442df130-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:37:21.516536 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.516487 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:21.516750 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.516553 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:21.529245 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.529211 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:21.596682 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.596645 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e52d1d8-5270-473b-8477-d1f1eda9fc7d" path="/var/lib/kubelet/pods/0e52d1d8-5270-473b-8477-d1f1eda9fc7d/volumes" Jul 07 19:37:21.604713 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.604683 2559 generic.go:356] "Generic (PLEG): container finished" podID="53f5529d-e435-47a4-96c6-0029442df130" containerID="88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6" exitCode=0 Jul 07 19:37:21.605145 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.604767 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" Jul 07 19:37:21.605145 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.604822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" event={"ID":"53f5529d-e435-47a4-96c6-0029442df130","Type":"ContainerDied","Data":"88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6"} Jul 07 19:37:21.605145 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.604869 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k" event={"ID":"53f5529d-e435-47a4-96c6-0029442df130","Type":"ContainerDied","Data":"a3cf214114089a3f0c2fe079a85c618e78984b12861e64f95b179563b994f0bb"} Jul 07 19:37:21.605145 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.604894 2559 scope.go:117] "RemoveContainer" containerID="88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6" Jul 07 19:37:21.614422 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.614403 2559 scope.go:117] "RemoveContainer" containerID="88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6" Jul 07 19:37:21.614723 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:37:21.614701 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6\": container with ID starting with 88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6 not found: ID does not exist" containerID="88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6" Jul 07 19:37:21.614801 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.614730 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6"} err="failed to get container status \"88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6\": rpc error: code = NotFound desc = could not find container \"88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6\": container with ID starting with 88f5ecb3a58afaa9563f9b4bf23e9b708574980260b494250409d8825fc4c2a6 not found: ID does not exist" Jul 07 19:37:21.617959 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.617932 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:37:21.623189 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.623146 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k"] Jul 07 19:37:21.628034 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:21.628008 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/auth-test-model-2620f1-kserve-59bd7d669f-qdz4k"] Jul 07 19:37:23.597136 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:23.597096 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53f5529d-e435-47a4-96c6-0029442df130" path="/var/lib/kubelet/pods/53f5529d-e435-47a4-96c6-0029442df130/volumes" Jul 07 19:37:45.084001 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.083965 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx"] Jul 07 19:37:45.084498 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.084282 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e52d1d8-5270-473b-8477-d1f1eda9fc7d" containerName="main" Jul 07 19:37:45.084498 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.084294 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e52d1d8-5270-473b-8477-d1f1eda9fc7d" containerName="main" Jul 07 19:37:45.084498 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.084310 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53f5529d-e435-47a4-96c6-0029442df130" containerName="main" Jul 07 19:37:45.084498 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.084316 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="53f5529d-e435-47a4-96c6-0029442df130" containerName="main" Jul 07 19:37:45.084498 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.084376 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e52d1d8-5270-473b-8477-d1f1eda9fc7d" containerName="main" Jul 07 19:37:45.084498 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.084383 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="53f5529d-e435-47a4-96c6-0029442df130" containerName="main" Jul 07 19:37:45.088602 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.088579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.091230 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.091198 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"openshift-service-ca.crt\"" Jul 07 19:37:45.092157 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.092137 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"kube-root-ca.crt\"" Jul 07 19:37:45.092270 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.092154 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"sub-test-model-2620f1-kserve-self-signed-certs\"" Jul 07 19:37:45.092270 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.092201 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"default-dockercfg-4dhwh\"" Jul 07 19:37:45.097813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.097783 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx"] Jul 07 19:37:45.208925 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.208885 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tls-certs\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.209125 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.208956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-dshm\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.209125 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.209012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-model-cache\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.209125 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.209060 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7fd\" (UniqueName: \"kubernetes.io/projected/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-kube-api-access-ms7fd\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.209125 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.209091 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-home\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.209387 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.209155 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tmp-dir\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.310539 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.310486 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7fd\" (UniqueName: \"kubernetes.io/projected/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-kube-api-access-ms7fd\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.310745 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.310627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-home\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.310745 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.310699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tmp-dir\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.310854 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.310771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tls-certs\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.310854 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.310843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-dshm\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.310953 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.310889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-model-cache\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.311141 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.311113 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-home\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.311347 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.311324 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-model-cache\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.311763 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.311740 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tmp-dir\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.315368 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.314996 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tls-certs\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.317663 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.317627 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-dshm\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.319548 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.319522 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7fd\" (UniqueName: \"kubernetes.io/projected/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-kube-api-access-ms7fd\") pod \"sub-test-model-2620f1-kserve-5bb544747c-dnrpx\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.398863 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.398804 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:45.543161 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.543132 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx"] Jul 07 19:37:45.545958 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:37:45.545924 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d6d1c3_e244_4b73_b7c7_3ed2b114862e.slice/crio-d3ef194c0410c5d4c2543a6cf272cd6f21a924410bcb2bee49c600d0182a2e17 WatchSource:0}: Error finding container d3ef194c0410c5d4c2543a6cf272cd6f21a924410bcb2bee49c600d0182a2e17: Status 404 returned error can't find the container with id d3ef194c0410c5d4c2543a6cf272cd6f21a924410bcb2bee49c600d0182a2e17 Jul 07 19:37:45.710256 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.710134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" event={"ID":"18d6d1c3-e244-4b73-b7c7-3ed2b114862e","Type":"ContainerStarted","Data":"9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0"} Jul 07 19:37:45.710256 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.710196 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" event={"ID":"18d6d1c3-e244-4b73-b7c7-3ed2b114862e","Type":"ContainerStarted","Data":"d3ef194c0410c5d4c2543a6cf272cd6f21a924410bcb2bee49c600d0182a2e17"} Jul 07 19:37:45.734246 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:45.734162 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" podStartSLOduration=0.734144258 podStartE2EDuration="734.144258ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:37:45.732947611 +0000 UTC m=+2190.754597618" watchObservedRunningTime="2026-07-07 19:37:45.734144258 +0000 UTC m=+2190.755794285" Jul 07 19:37:55.399213 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:55.399157 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:55.399213 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:55.399228 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:55.412087 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:55.412057 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:37:55.764587 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:37:55.764503 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:38:18.976293 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:18.976255 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2"] Jul 07 19:38:20.093594 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:20.093557 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx"] Jul 07 19:38:24.506570 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.506529 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq"] Jul 07 19:38:24.509804 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.509785 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.512354 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.512334 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c576d2\"/\"rate-test-model-c576d2-kserve-self-signed-certs\"" Jul 07 19:38:24.520773 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.520745 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq"] Jul 07 19:38:24.546854 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.546819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-dshm\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.547041 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.546875 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrpks\" (UniqueName: \"kubernetes.io/projected/8703dfb1-9d62-4da2-9acd-bb37d545b211-kube-api-access-rrpks\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.547041 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.546941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-home\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.547041 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.547005 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-tmp-dir\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.547204 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.547078 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-model-cache\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.547204 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.547113 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8703dfb1-9d62-4da2-9acd-bb37d545b211-tls-certs\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.648423 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.648379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-dshm\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.648629 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.648464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrpks\" (UniqueName: \"kubernetes.io/projected/8703dfb1-9d62-4da2-9acd-bb37d545b211-kube-api-access-rrpks\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.648719 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.648664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-home\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.648777 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.648721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-tmp-dir\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.648839 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.648808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-model-cache\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.648893 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.648847 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8703dfb1-9d62-4da2-9acd-bb37d545b211-tls-certs\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.649058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.648990 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-home\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.649058 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.649027 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-tmp-dir\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.649243 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.649216 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-model-cache\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.650719 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.650698 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-dshm\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.651108 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.651091 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8703dfb1-9d62-4da2-9acd-bb37d545b211-tls-certs\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.657403 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.657371 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrpks\" (UniqueName: \"kubernetes.io/projected/8703dfb1-9d62-4da2-9acd-bb37d545b211-kube-api-access-rrpks\") pod \"rate-test-model-c576d2-kserve-ff9bbb95-6k7tq\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.819384 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.819251 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:24.962408 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:24.962369 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq"] Jul 07 19:38:24.965328 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:38:24.965297 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8703dfb1_9d62_4da2_9acd_bb37d545b211.slice/crio-51e0928bfd9b8bc1c8b1fcba7978bdd513cbc2709f871bab3d1620580f51ed66 WatchSource:0}: Error finding container 51e0928bfd9b8bc1c8b1fcba7978bdd513cbc2709f871bab3d1620580f51ed66: Status 404 returned error can't find the container with id 51e0928bfd9b8bc1c8b1fcba7978bdd513cbc2709f871bab3d1620580f51ed66 Jul 07 19:38:25.888536 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:25.888498 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" event={"ID":"8703dfb1-9d62-4da2-9acd-bb37d545b211","Type":"ContainerStarted","Data":"afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009"} Jul 07 19:38:25.888536 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:25.888542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" event={"ID":"8703dfb1-9d62-4da2-9acd-bb37d545b211","Type":"ContainerStarted","Data":"51e0928bfd9b8bc1c8b1fcba7978bdd513cbc2709f871bab3d1620580f51ed66"} Jul 07 19:38:25.910530 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:25.910478 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" podStartSLOduration=1.9104596169999999 podStartE2EDuration="1.910459617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:38:25.909555613 +0000 UTC m=+2230.931205617" watchObservedRunningTime="2026-07-07 19:38:25.910459617 +0000 UTC m=+2230.932109621" Jul 07 19:38:33.995819 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:33.995778 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" podUID="cdef644b-62e4-4383-9268-066bb8f0abf2" containerName="main" containerID="cri-o://d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6" gracePeriod=45 Jul 07 19:38:34.253433 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.253352 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:38:34.337520 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337481 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-model-cache\") pod \"cdef644b-62e4-4383-9268-066bb8f0abf2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " Jul 07 19:38:34.337718 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337548 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjgq6\" (UniqueName: \"kubernetes.io/projected/cdef644b-62e4-4383-9268-066bb8f0abf2-kube-api-access-xjgq6\") pod \"cdef644b-62e4-4383-9268-066bb8f0abf2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " Jul 07 19:38:34.337718 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337574 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-home\") pod \"cdef644b-62e4-4383-9268-066bb8f0abf2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " Jul 07 19:38:34.337718 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337589 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-tmp-dir\") pod \"cdef644b-62e4-4383-9268-066bb8f0abf2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " Jul 07 19:38:34.337718 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337652 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-dshm\") pod \"cdef644b-62e4-4383-9268-066bb8f0abf2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " Jul 07 19:38:34.337718 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337711 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdef644b-62e4-4383-9268-066bb8f0abf2-tls-certs\") pod \"cdef644b-62e4-4383-9268-066bb8f0abf2\" (UID: \"cdef644b-62e4-4383-9268-066bb8f0abf2\") " Jul 07 19:38:34.337983 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337796 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdef644b-62e4-4383-9268-066bb8f0abf2" (UID: "cdef644b-62e4-4383-9268-066bb8f0abf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:34.337983 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337914 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-home" (OuterVolumeSpecName: "home") pod "cdef644b-62e4-4383-9268-066bb8f0abf2" (UID: "cdef644b-62e4-4383-9268-066bb8f0abf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:34.337983 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.337920 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdef644b-62e4-4383-9268-066bb8f0abf2" (UID: "cdef644b-62e4-4383-9268-066bb8f0abf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:34.338101 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.338012 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:34.338101 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.338032 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:34.338101 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.338044 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:34.339920 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.339891 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdef644b-62e4-4383-9268-066bb8f0abf2-kube-api-access-xjgq6" (OuterVolumeSpecName: "kube-api-access-xjgq6") pod "cdef644b-62e4-4383-9268-066bb8f0abf2" (UID: "cdef644b-62e4-4383-9268-066bb8f0abf2"). InnerVolumeSpecName "kube-api-access-xjgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:38:34.340000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.339892 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdef644b-62e4-4383-9268-066bb8f0abf2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdef644b-62e4-4383-9268-066bb8f0abf2" (UID: "cdef644b-62e4-4383-9268-066bb8f0abf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:38:34.340000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.339935 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-dshm" (OuterVolumeSpecName: "dshm") pod "cdef644b-62e4-4383-9268-066bb8f0abf2" (UID: "cdef644b-62e4-4383-9268-066bb8f0abf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:34.438938 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.438895 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjgq6\" (UniqueName: \"kubernetes.io/projected/cdef644b-62e4-4383-9268-066bb8f0abf2-kube-api-access-xjgq6\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:34.438938 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.438932 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdef644b-62e4-4383-9268-066bb8f0abf2-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:34.438938 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.438942 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdef644b-62e4-4383-9268-066bb8f0abf2-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:34.819834 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.819793 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:34.819834 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.819840 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:34.832487 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.832458 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:34.927421 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.927383 2559 generic.go:356] "Generic (PLEG): container finished" podID="cdef644b-62e4-4383-9268-066bb8f0abf2" containerID="d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6" exitCode=0 Jul 07 19:38:34.927622 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.927430 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" Jul 07 19:38:34.927622 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.927472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" event={"ID":"cdef644b-62e4-4383-9268-066bb8f0abf2","Type":"ContainerDied","Data":"d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6"} Jul 07 19:38:34.927622 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.927515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2" event={"ID":"cdef644b-62e4-4383-9268-066bb8f0abf2","Type":"ContainerDied","Data":"d0003f802f2172373b111cfa53b48e899b4d56f26cd3527e6e642b6d7463acbc"} Jul 07 19:38:34.927622 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.927541 2559 scope.go:117] "RemoveContainer" containerID="d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6" Jul 07 19:38:34.936701 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.936678 2559 scope.go:117] "RemoveContainer" containerID="d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6" Jul 07 19:38:34.937025 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:38:34.937006 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6\": container with ID starting with d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6 not found: ID does not exist" containerID="d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6" Jul 07 19:38:34.937077 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.937035 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6"} err="failed to get container status \"d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6\": rpc error: code = NotFound desc = could not find container \"d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6\": container with ID starting with d89641afd7b567243ee9594db3e8f409fb06be954d5f162409387fadbb37c9d6 not found: ID does not exist" Jul 07 19:38:34.940527 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.940498 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:38:34.952182 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.952128 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2"] Jul 07 19:38:34.958028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:34.957995 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/sub-test-model-c576d2-kserve-766d87645-vfpl2"] Jul 07 19:38:35.113315 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.113273 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" podUID="18d6d1c3-e244-4b73-b7c7-3ed2b114862e" containerName="main" containerID="cri-o://9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0" gracePeriod=45 Jul 07 19:38:35.371804 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.371724 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:38:35.448882 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.448840 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7fd\" (UniqueName: \"kubernetes.io/projected/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-kube-api-access-ms7fd\") pod \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " Jul 07 19:38:35.449078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.448894 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-dshm\") pod \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " Jul 07 19:38:35.449078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.448915 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-model-cache\") pod \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " Jul 07 19:38:35.449078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.448957 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-home\") pod \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " Jul 07 19:38:35.449078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.448981 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tls-certs\") pod \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " Jul 07 19:38:35.449078 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.449009 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tmp-dir\") pod \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\" (UID: \"18d6d1c3-e244-4b73-b7c7-3ed2b114862e\") " Jul 07 19:38:35.449363 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.449277 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-home" (OuterVolumeSpecName: "home") pod "18d6d1c3-e244-4b73-b7c7-3ed2b114862e" (UID: "18d6d1c3-e244-4b73-b7c7-3ed2b114862e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:35.449363 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.449284 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-model-cache" (OuterVolumeSpecName: "model-cache") pod "18d6d1c3-e244-4b73-b7c7-3ed2b114862e" (UID: "18d6d1c3-e244-4b73-b7c7-3ed2b114862e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:35.449436 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.449405 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18d6d1c3-e244-4b73-b7c7-3ed2b114862e" (UID: "18d6d1c3-e244-4b73-b7c7-3ed2b114862e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:35.451297 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.451250 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-dshm" (OuterVolumeSpecName: "dshm") pod "18d6d1c3-e244-4b73-b7c7-3ed2b114862e" (UID: "18d6d1c3-e244-4b73-b7c7-3ed2b114862e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:38:35.451297 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.451279 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-kube-api-access-ms7fd" (OuterVolumeSpecName: "kube-api-access-ms7fd") pod "18d6d1c3-e244-4b73-b7c7-3ed2b114862e" (UID: "18d6d1c3-e244-4b73-b7c7-3ed2b114862e"). InnerVolumeSpecName "kube-api-access-ms7fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:38:35.451297 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.451251 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18d6d1c3-e244-4b73-b7c7-3ed2b114862e" (UID: "18d6d1c3-e244-4b73-b7c7-3ed2b114862e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:38:35.549757 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.549710 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:35.549757 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.549748 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:35.549757 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.549765 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:35.550059 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.549777 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7fd\" (UniqueName: \"kubernetes.io/projected/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-kube-api-access-ms7fd\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:35.550059 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.549790 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:35.550059 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.549801 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d6d1c3-e244-4b73-b7c7-3ed2b114862e-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:38:35.597312 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.597278 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdef644b-62e4-4383-9268-066bb8f0abf2" path="/var/lib/kubelet/pods/cdef644b-62e4-4383-9268-066bb8f0abf2/volumes" Jul 07 19:38:35.933223 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.933158 2559 generic.go:356] "Generic (PLEG): container finished" podID="18d6d1c3-e244-4b73-b7c7-3ed2b114862e" containerID="9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0" exitCode=0 Jul 07 19:38:35.933223 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.933228 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" Jul 07 19:38:35.933505 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.933275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" event={"ID":"18d6d1c3-e244-4b73-b7c7-3ed2b114862e","Type":"ContainerDied","Data":"9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0"} Jul 07 19:38:35.933505 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.933321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx" event={"ID":"18d6d1c3-e244-4b73-b7c7-3ed2b114862e","Type":"ContainerDied","Data":"d3ef194c0410c5d4c2543a6cf272cd6f21a924410bcb2bee49c600d0182a2e17"} Jul 07 19:38:35.933505 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.933345 2559 scope.go:117] "RemoveContainer" containerID="9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0" Jul 07 19:38:35.942302 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.942276 2559 scope.go:117] "RemoveContainer" containerID="9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0" Jul 07 19:38:35.942605 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:38:35.942583 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0\": container with ID starting with 9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0 not found: ID does not exist" containerID="9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0" Jul 07 19:38:35.942645 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.942615 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0"} err="failed to get container status \"9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0\": rpc error: code = NotFound desc = could not find container \"9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0\": container with ID starting with 9391b8cb3ad6b2698cb5bccf4c5e2ec90c2da69a3a4cf757585512081b0fbcf0 not found: ID does not exist" Jul 07 19:38:35.951376 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.951340 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx"] Jul 07 19:38:35.954939 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:35.954906 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/sub-test-model-2620f1-kserve-5bb544747c-dnrpx"] Jul 07 19:38:37.596525 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:38:37.596430 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18d6d1c3-e244-4b73-b7c7-3ed2b114862e" path="/var/lib/kubelet/pods/18d6d1c3-e244-4b73-b7c7-3ed2b114862e/volumes" Jul 07 19:39:00.702503 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.702467 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm"] Jul 07 19:39:00.703109 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.702927 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d6d1c3-e244-4b73-b7c7-3ed2b114862e" containerName="main" Jul 07 19:39:00.703109 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.702946 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d6d1c3-e244-4b73-b7c7-3ed2b114862e" containerName="main" Jul 07 19:39:00.703109 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.702965 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdef644b-62e4-4383-9268-066bb8f0abf2" containerName="main" Jul 07 19:39:00.703109 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.702974 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdef644b-62e4-4383-9268-066bb8f0abf2" containerName="main" Jul 07 19:39:00.703109 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.703062 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d6d1c3-e244-4b73-b7c7-3ed2b114862e" containerName="main" Jul 07 19:39:00.703109 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.703074 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdef644b-62e4-4383-9268-066bb8f0abf2" containerName="main" Jul 07 19:39:00.706181 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.706141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.711997 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.711966 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"openshift-service-ca.crt\"" Jul 07 19:39:00.712235 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.712025 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"default-dockercfg-4dhwh\"" Jul 07 19:39:00.713329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.712908 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"kube-root-ca.crt\"" Jul 07 19:39:00.713329 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.713066 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2620f1\"/\"rate-test-model-2620f1-kserve-self-signed-certs\"" Jul 07 19:39:00.717518 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.717490 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm"] Jul 07 19:39:00.879770 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.879730 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-model-cache\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.879770 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.879773 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/6a72c7df-9fa9-4ced-a60b-84d19c37e175-kube-api-access-wsx7d\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.880000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.879858 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-home\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.880000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.879931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tls-certs\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.880000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.879959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-dshm\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.880000 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.879988 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tmp-dir\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.980821 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.980709 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tmp-dir\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.980821 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.980812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-model-cache\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.981054 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.980840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/6a72c7df-9fa9-4ced-a60b-84d19c37e175-kube-api-access-wsx7d\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.981054 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.980972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-home\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.981054 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.981009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tls-certs\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.981054 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.981038 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-dshm\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.981282 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.981203 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tmp-dir\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.981335 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.981283 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-model-cache\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.981870 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.981845 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-home\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.983347 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.983320 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-dshm\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.983751 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.983731 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tls-certs\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:00.991228 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:00.991196 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/6a72c7df-9fa9-4ced-a60b-84d19c37e175-kube-api-access-wsx7d\") pod \"rate-test-model-2620f1-kserve-54b5478b7-2fznm\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:01.015933 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:01.015894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:01.159796 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:01.159758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm"] Jul 07 19:39:01.161665 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:39:01.161637 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a72c7df_9fa9_4ced_a60b_84d19c37e175.slice/crio-cbfd3f30f800b1dc77c4702e4bd09ee67a5b01deb4dd3e24c9f0bdc8e2599c40 WatchSource:0}: Error finding container cbfd3f30f800b1dc77c4702e4bd09ee67a5b01deb4dd3e24c9f0bdc8e2599c40: Status 404 returned error can't find the container with id cbfd3f30f800b1dc77c4702e4bd09ee67a5b01deb4dd3e24c9f0bdc8e2599c40 Jul 07 19:39:02.045254 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:02.045211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" event={"ID":"6a72c7df-9fa9-4ced-a60b-84d19c37e175","Type":"ContainerStarted","Data":"072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2"} Jul 07 19:39:02.045254 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:02.045260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" event={"ID":"6a72c7df-9fa9-4ced-a60b-84d19c37e175","Type":"ContainerStarted","Data":"cbfd3f30f800b1dc77c4702e4bd09ee67a5b01deb4dd3e24c9f0bdc8e2599c40"} Jul 07 19:39:02.077377 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:02.077321 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" podStartSLOduration=2.077305356 podStartE2EDuration="2.077305356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 19:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:39:02.076216827 +0000 UTC m=+2267.097866832" watchObservedRunningTime="2026-07-07 19:39:02.077305356 +0000 UTC m=+2267.098955359" Jul 07 19:39:11.017024 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:11.016977 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:11.017024 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:11.017036 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:11.029927 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:11.029899 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:11.095303 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:11.095267 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:39:51.909251 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:51.909198 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq"] Jul 07 19:39:52.974844 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:39:52.974806 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm"] Jul 07 19:40:06.930384 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:06.930263 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" podUID="8703dfb1-9d62-4da2-9acd-bb37d545b211" containerName="main" containerID="cri-o://afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009" gracePeriod=45 Jul 07 19:40:07.180028 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.180002 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:40:07.264389 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264293 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-dshm\") pod \"8703dfb1-9d62-4da2-9acd-bb37d545b211\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " Jul 07 19:40:07.264389 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264336 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-tmp-dir\") pod \"8703dfb1-9d62-4da2-9acd-bb37d545b211\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " Jul 07 19:40:07.264389 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264363 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-model-cache\") pod \"8703dfb1-9d62-4da2-9acd-bb37d545b211\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " Jul 07 19:40:07.264686 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8703dfb1-9d62-4da2-9acd-bb37d545b211-tls-certs\") pod \"8703dfb1-9d62-4da2-9acd-bb37d545b211\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " Jul 07 19:40:07.264686 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264438 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-home\") pod \"8703dfb1-9d62-4da2-9acd-bb37d545b211\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " Jul 07 19:40:07.264686 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264487 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrpks\" (UniqueName: \"kubernetes.io/projected/8703dfb1-9d62-4da2-9acd-bb37d545b211-kube-api-access-rrpks\") pod \"8703dfb1-9d62-4da2-9acd-bb37d545b211\" (UID: \"8703dfb1-9d62-4da2-9acd-bb37d545b211\") " Jul 07 19:40:07.264686 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264624 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8703dfb1-9d62-4da2-9acd-bb37d545b211" (UID: "8703dfb1-9d62-4da2-9acd-bb37d545b211"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:07.264843 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264691 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-model-cache" (OuterVolumeSpecName: "model-cache") pod "8703dfb1-9d62-4da2-9acd-bb37d545b211" (UID: "8703dfb1-9d62-4da2-9acd-bb37d545b211"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:07.264843 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264728 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:07.264843 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.264747 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-home" (OuterVolumeSpecName: "home") pod "8703dfb1-9d62-4da2-9acd-bb37d545b211" (UID: "8703dfb1-9d62-4da2-9acd-bb37d545b211"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:07.266588 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.266566 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-dshm" (OuterVolumeSpecName: "dshm") pod "8703dfb1-9d62-4da2-9acd-bb37d545b211" (UID: "8703dfb1-9d62-4da2-9acd-bb37d545b211"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:07.266588 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.266572 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8703dfb1-9d62-4da2-9acd-bb37d545b211-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8703dfb1-9d62-4da2-9acd-bb37d545b211" (UID: "8703dfb1-9d62-4da2-9acd-bb37d545b211"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:40:07.266711 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.266584 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8703dfb1-9d62-4da2-9acd-bb37d545b211-kube-api-access-rrpks" (OuterVolumeSpecName: "kube-api-access-rrpks") pod "8703dfb1-9d62-4da2-9acd-bb37d545b211" (UID: "8703dfb1-9d62-4da2-9acd-bb37d545b211"). InnerVolumeSpecName "kube-api-access-rrpks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:40:07.319805 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.319770 2559 generic.go:356] "Generic (PLEG): container finished" podID="8703dfb1-9d62-4da2-9acd-bb37d545b211" containerID="afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009" exitCode=0 Jul 07 19:40:07.319973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.319813 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" Jul 07 19:40:07.319973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.319856 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" event={"ID":"8703dfb1-9d62-4da2-9acd-bb37d545b211","Type":"ContainerDied","Data":"afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009"} Jul 07 19:40:07.319973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.319904 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq" event={"ID":"8703dfb1-9d62-4da2-9acd-bb37d545b211","Type":"ContainerDied","Data":"51e0928bfd9b8bc1c8b1fcba7978bdd513cbc2709f871bab3d1620580f51ed66"} Jul 07 19:40:07.319973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.319926 2559 scope.go:117] "RemoveContainer" containerID="afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009" Jul 07 19:40:07.328913 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.328888 2559 scope.go:117] "RemoveContainer" containerID="afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009" Jul 07 19:40:07.329229 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:40:07.329207 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009\": container with ID starting with afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009 not found: ID does not exist" containerID="afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009" Jul 07 19:40:07.329290 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.329241 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009"} err="failed to get container status \"afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009\": rpc error: code = NotFound desc = could not find container \"afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009\": container with ID starting with afc9010ecbb90569cb463e852530eae61d5d4e0c0b9003cc8f45169615392009 not found: ID does not exist" Jul 07 19:40:07.341773 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.341736 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq"] Jul 07 19:40:07.348200 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.348158 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c576d2/rate-test-model-c576d2-kserve-ff9bbb95-6k7tq"] Jul 07 19:40:07.365888 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.365862 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:07.365888 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.365887 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrpks\" (UniqueName: \"kubernetes.io/projected/8703dfb1-9d62-4da2-9acd-bb37d545b211-kube-api-access-rrpks\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:07.366018 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.365896 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:07.366018 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.365907 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8703dfb1-9d62-4da2-9acd-bb37d545b211-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:07.366018 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.365916 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8703dfb1-9d62-4da2-9acd-bb37d545b211-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:07.595953 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.595916 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8703dfb1-9d62-4da2-9acd-bb37d545b211" path="/var/lib/kubelet/pods/8703dfb1-9d62-4da2-9acd-bb37d545b211/volumes" Jul 07 19:40:07.995478 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:07.995388 2559 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" podUID="6a72c7df-9fa9-4ced-a60b-84d19c37e175" containerName="main" containerID="cri-o://072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2" gracePeriod=45 Jul 07 19:40:08.253753 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.253679 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:40:08.326898 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.326864 2559 generic.go:356] "Generic (PLEG): container finished" podID="6a72c7df-9fa9-4ced-a60b-84d19c37e175" containerID="072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2" exitCode=0 Jul 07 19:40:08.327083 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.326905 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" Jul 07 19:40:08.327083 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.326946 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" event={"ID":"6a72c7df-9fa9-4ced-a60b-84d19c37e175","Type":"ContainerDied","Data":"072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2"} Jul 07 19:40:08.327083 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.326992 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm" event={"ID":"6a72c7df-9fa9-4ced-a60b-84d19c37e175","Type":"ContainerDied","Data":"cbfd3f30f800b1dc77c4702e4bd09ee67a5b01deb4dd3e24c9f0bdc8e2599c40"} Jul 07 19:40:08.327083 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.327013 2559 scope.go:117] "RemoveContainer" containerID="072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2" Jul 07 19:40:08.335969 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.335945 2559 scope.go:117] "RemoveContainer" containerID="072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2" Jul 07 19:40:08.336318 ip-10-0-137-8 kubenswrapper[2559]: E0707 19:40:08.336298 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2\": container with ID starting with 072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2 not found: ID does not exist" containerID="072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2" Jul 07 19:40:08.336372 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.336328 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2"} err="failed to get container status \"072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2\": rpc error: code = NotFound desc = could not find container \"072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2\": container with ID starting with 072200194fe016a74f9a05c001237c49bfdbb5a54e6837d222e4b08d8a75d3a2 not found: ID does not exist" Jul 07 19:40:08.375576 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375538 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tls-certs\") pod \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " Jul 07 19:40:08.375787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375604 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tmp-dir\") pod \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " Jul 07 19:40:08.375787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375631 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/6a72c7df-9fa9-4ced-a60b-84d19c37e175-kube-api-access-wsx7d\") pod \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " Jul 07 19:40:08.375787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375671 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-model-cache\") pod \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " Jul 07 19:40:08.375787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-dshm\") pod \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " Jul 07 19:40:08.375787 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375747 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-home\") pod \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\" (UID: \"6a72c7df-9fa9-4ced-a60b-84d19c37e175\") " Jul 07 19:40:08.376037 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375955 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a72c7df-9fa9-4ced-a60b-84d19c37e175" (UID: "6a72c7df-9fa9-4ced-a60b-84d19c37e175"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:08.376037 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.375971 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a72c7df-9fa9-4ced-a60b-84d19c37e175" (UID: "6a72c7df-9fa9-4ced-a60b-84d19c37e175"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:08.376121 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.376097 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-home" (OuterVolumeSpecName: "home") pod "6a72c7df-9fa9-4ced-a60b-84d19c37e175" (UID: "6a72c7df-9fa9-4ced-a60b-84d19c37e175"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:08.377912 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.377876 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-dshm" (OuterVolumeSpecName: "dshm") pod "6a72c7df-9fa9-4ced-a60b-84d19c37e175" (UID: "6a72c7df-9fa9-4ced-a60b-84d19c37e175"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 19:40:08.378033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.377930 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a72c7df-9fa9-4ced-a60b-84d19c37e175-kube-api-access-wsx7d" (OuterVolumeSpecName: "kube-api-access-wsx7d") pod "6a72c7df-9fa9-4ced-a60b-84d19c37e175" (UID: "6a72c7df-9fa9-4ced-a60b-84d19c37e175"). InnerVolumeSpecName "kube-api-access-wsx7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 19:40:08.378033 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.377934 2559 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a72c7df-9fa9-4ced-a60b-84d19c37e175" (UID: "6a72c7df-9fa9-4ced-a60b-84d19c37e175"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 19:40:08.477443 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.477397 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-home\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:08.477443 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.477432 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tls-certs\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:08.477443 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.477444 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-tmp-dir\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:08.477443 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.477453 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsx7d\" (UniqueName: \"kubernetes.io/projected/6a72c7df-9fa9-4ced-a60b-84d19c37e175-kube-api-access-wsx7d\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:08.477712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.477465 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-model-cache\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:08.477712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.477473 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a72c7df-9fa9-4ced-a60b-84d19c37e175-dshm\") on node \"ip-10-0-137-8.ec2.internal\" DevicePath \"\"" Jul 07 19:40:08.650327 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.650285 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm"] Jul 07 19:40:08.656713 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:08.656677 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2620f1/rate-test-model-2620f1-kserve-54b5478b7-2fznm"] Jul 07 19:40:09.596558 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:09.596521 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a72c7df-9fa9-4ced-a60b-84d19c37e175" path="/var/lib/kubelet/pods/6a72c7df-9fa9-4ced-a60b-84d19c37e175/volumes" Jul 07 19:40:51.694869 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:51.694831 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wbp6w_e672aeec-2a0a-45f0-b295-70412ac69d22/manager/0.log" Jul 07 19:40:51.811493 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:51.811467 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59996cbd9d-4bz6w_1cf01b44-67ab-400c-8d26-9da703350f0d/manager/0.log" Jul 07 19:40:51.819667 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:51.819639 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59996cbd9d-4bz6w_1cf01b44-67ab-400c-8d26-9da703350f0d/copy-manifests/0.log" Jul 07 19:40:54.660419 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:54.660384 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45k75z_261cf77c-43a3-4f06-9c0d-8ea9b666615e/istio-proxy/0.log" Jul 07 19:40:55.013759 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:55.013679 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-955bd7587-mp5tm_458762dd-80a2-44a9-a6f0-fd1918ed4a70/kube-auth-proxy/0.log" Jul 07 19:40:55.137764 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:55.137732 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62l8c_7c3626d9-37a5-4954-bca8-671444c66047/istio-proxy/1.log" Jul 07 19:40:55.851901 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:55.851867 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-g9jhc_0a321d62-4f1f-4e34-9953-5b3e40ad5b52/main/0.log" Jul 07 19:40:56.202214 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:56.202117 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbg7b67_14992779-c28a-4285-8b65-904b8efaf99d/main/0.log" Jul 07 19:40:56.317813 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:56.317781 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5854r_1928cd31-a85d-4235-937c-66944ccf2fed/main/0.log" Jul 07 19:40:56.428567 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:40:56.428540 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-njhfh_88cc5a57-070e-4909-941d-f5f37c0525f1/main/0.log" Jul 07 19:41:04.236363 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:04.236331 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qfr4x_3f9aebed-5381-4ead-9371-161f4168b64a/global-pull-secret-syncer/0.log" Jul 07 19:41:04.380971 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:04.380943 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sdhn8_7d48bec1-d448-457f-922e-e75c8de4a3a4/konnectivity-agent/0.log" Jul 07 19:41:04.493325 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:04.493244 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-8.ec2.internal_2ca19b4fae04ee4ffd4ef9397655850d/haproxy/0.log" Jul 07 19:41:12.645125 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:12.645095 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd6pq_015fa7a7-c259-40ff-9404-c9d00bcb1e17/node-exporter/0.log" Jul 07 19:41:12.677569 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:12.677535 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd6pq_015fa7a7-c259-40ff-9404-c9d00bcb1e17/kube-rbac-proxy/0.log" Jul 07 19:41:12.703707 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:12.703681 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nd6pq_015fa7a7-c259-40ff-9404-c9d00bcb1e17/init-textfile/0.log" Jul 07 19:41:14.191426 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.191391 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq"] Jul 07 19:41:14.191811 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.191696 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a72c7df-9fa9-4ced-a60b-84d19c37e175" containerName="main" Jul 07 19:41:14.191811 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.191708 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a72c7df-9fa9-4ced-a60b-84d19c37e175" containerName="main" Jul 07 19:41:14.191811 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.191716 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8703dfb1-9d62-4da2-9acd-bb37d545b211" containerName="main" Jul 07 19:41:14.191811 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.191721 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8703dfb1-9d62-4da2-9acd-bb37d545b211" containerName="main" Jul 07 19:41:14.191811 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.191783 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a72c7df-9fa9-4ced-a60b-84d19c37e175" containerName="main" Jul 07 19:41:14.191811 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.191793 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8703dfb1-9d62-4da2-9acd-bb37d545b211" containerName="main" Jul 07 19:41:14.194652 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.194633 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.198488 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.198461 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v4c88\"/\"openshift-service-ca.crt\"" Jul 07 19:41:14.198668 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.198534 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v4c88\"/\"default-dockercfg-drvns\"" Jul 07 19:41:14.199677 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.199655 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v4c88\"/\"kube-root-ca.crt\"" Jul 07 19:41:14.206802 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.206768 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-podres\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.206802 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.206807 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-lib-modules\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.206997 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.206926 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-sys\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.206997 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.206956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-proc\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.206997 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.206979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvw28\" (UniqueName: \"kubernetes.io/projected/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-kube-api-access-gvw28\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.210472 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.210440 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq"] Jul 07 19:41:14.308433 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-sys\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308433 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-proc\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308454 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvw28\" (UniqueName: \"kubernetes.io/projected/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-kube-api-access-gvw28\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-podres\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-lib-modules\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308527 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-proc\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308528 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-sys\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308619 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-podres\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.308644 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.308629 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-lib-modules\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.319022 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.318981 2559 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvw28\" (UniqueName: \"kubernetes.io/projected/f7fe760a-18e5-48b2-89ff-8f0e58ddbea9-kube-api-access-gvw28\") pod \"perf-node-gather-daemonset-jp9kq\" (UID: \"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9\") " pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.505015 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.504971 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:14.639973 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:14.639945 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq"] Jul 07 19:41:14.642687 ip-10-0-137-8 kubenswrapper[2559]: W0707 19:41:14.642655 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf7fe760a_18e5_48b2_89ff_8f0e58ddbea9.slice/crio-97378f37b57f546e7da3b6553cec1888957af4249c2042bcc3a7996c86a85407 WatchSource:0}: Error finding container 97378f37b57f546e7da3b6553cec1888957af4249c2042bcc3a7996c86a85407: Status 404 returned error can't find the container with id 97378f37b57f546e7da3b6553cec1888957af4249c2042bcc3a7996c86a85407 Jul 07 19:41:15.620024 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:15.619992 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" event={"ID":"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9","Type":"ContainerStarted","Data":"82012dbac6a0398c7d12d6515c70a3b681a80b23f8691f132644cc362900df55"} Jul 07 19:41:15.620447 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:15.620044 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" event={"ID":"f7fe760a-18e5-48b2-89ff-8f0e58ddbea9","Type":"ContainerStarted","Data":"97378f37b57f546e7da3b6553cec1888957af4249c2042bcc3a7996c86a85407"} Jul 07 19:41:15.620447 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:15.620117 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:15.639110 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:15.639053 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" podStartSLOduration=1.6390344479999999 podStartE2EDuration="1.639034448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 19:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 19:41:15.63755794 +0000 UTC m=+2400.659207944" watchObservedRunningTime="2026-07-07 19:41:15.639034448 +0000 UTC m=+2400.660684451" Jul 07 19:41:15.673618 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:15.673499 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:41:15.687935 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:15.677904 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tbbkn_d9e0d672-2ba3-4061-83d2-d7a12a98e1ff/ovn-acl-logging/0.log" Jul 07 19:41:17.696441 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:17.696415 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cpnwl_fb0a3509-52f2-4f00-b988-b1402d0e916c/dns/0.log" Jul 07 19:41:17.721075 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:17.721045 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cpnwl_fb0a3509-52f2-4f00-b988-b1402d0e916c/kube-rbac-proxy/0.log" Jul 07 19:41:17.826124 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:17.826093 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6phhb_f399a49f-d298-40ef-97b5-8dff9127f599/dns-node-resolver/0.log" Jul 07 19:41:18.525090 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:18.525055 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ltm4_ce283c41-58d4-4fe9-af91-4a005dd13dcb/node-ca/0.log" Jul 07 19:41:19.556519 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:19.556487 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45k75z_261cf77c-43a3-4f06-9c0d-8ea9b666615e/istio-proxy/0.log" Jul 07 19:41:19.724842 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:19.724813 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-955bd7587-mp5tm_458762dd-80a2-44a9-a6f0-fd1918ed4a70/kube-auth-proxy/0.log" Jul 07 19:41:19.758712 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:19.758683 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62l8c_7c3626d9-37a5-4954-bca8-671444c66047/istio-proxy/0.log" Jul 07 19:41:19.785704 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:19.785670 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62l8c_7c3626d9-37a5-4954-bca8-671444c66047/istio-proxy/1.log" Jul 07 19:41:20.806560 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:20.806532 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-slglq_77480f2d-0e6a-484f-acff-a713190d7e04/serve-healthcheck-canary/0.log" Jul 07 19:41:21.555398 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:21.555373 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fhjbt_c68e489b-4755-42fe-a811-119b84285d00/kube-rbac-proxy/0.log" Jul 07 19:41:21.583863 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:21.583836 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fhjbt_c68e489b-4755-42fe-a811-119b84285d00/exporter/0.log" Jul 07 19:41:21.609546 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:21.609520 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fhjbt_c68e489b-4755-42fe-a811-119b84285d00/extractor/0.log" Jul 07 19:41:21.633529 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:21.633503 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v4c88/perf-node-gather-daemonset-jp9kq" Jul 07 19:41:24.545750 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:24.545714 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wbp6w_e672aeec-2a0a-45f0-b295-70412ac69d22/manager/0.log" Jul 07 19:41:24.638607 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:24.638575 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59996cbd9d-4bz6w_1cf01b44-67ab-400c-8d26-9da703350f0d/manager/0.log" Jul 07 19:41:24.717489 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:24.717458 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-59996cbd9d-4bz6w_1cf01b44-67ab-400c-8d26-9da703350f0d/copy-manifests/0.log" Jul 07 19:41:25.072262 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:25.072037 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45k75z_261cf77c-43a3-4f06-9c0d-8ea9b666615e/istio-proxy/0.log" Jul 07 19:41:25.193077 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:25.193047 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-955bd7587-mp5tm_458762dd-80a2-44a9-a6f0-fd1918ed4a70/kube-auth-proxy/0.log" Jul 07 19:41:25.226924 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:25.226888 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62l8c_7c3626d9-37a5-4954-bca8-671444c66047/istio-proxy/0.log" Jul 07 19:41:25.251580 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:25.251546 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62l8c_7c3626d9-37a5-4954-bca8-671444c66047/istio-proxy/1.log" Jul 07 19:41:26.225174 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:26.225133 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6c8bfdd549-5nzsx_209bd0c1-fade-461e-a4cf-ac5eeb85cac2/manager/0.log" Jul 07 19:41:26.663139 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:26.663111 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf45k75z_261cf77c-43a3-4f06-9c0d-8ea9b666615e/istio-proxy/0.log" Jul 07 19:41:26.792892 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:26.792864 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-955bd7587-mp5tm_458762dd-80a2-44a9-a6f0-fd1918ed4a70/kube-auth-proxy/0.log" Jul 07 19:41:26.813200 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:26.813148 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62l8c_7c3626d9-37a5-4954-bca8-671444c66047/istio-proxy/0.log" Jul 07 19:41:26.846606 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:26.846567 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62l8c_7c3626d9-37a5-4954-bca8-671444c66047/istio-proxy/1.log" Jul 07 19:41:31.319910 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:31.319875 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-qpkf6_ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c/migrator/0.log" Jul 07 19:41:31.344485 ip-10-0-137-8 kubenswrapper[2559]: I0707 19:41:31.344450 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-qpkf6_ac4d495e-7ff3-4c4b-a29a-c47e7e64e65c/graceful-termination/0.log"