Sep 16 18:42:36.026843 ip-10-0-138-79 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 18:42:36.026854 ip-10-0-138-79 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 18:42:36.026863 ip-10-0-138-79 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 18:42:36.027176 ip-10-0-138-79 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 18:42:46.238015 ip-10-0-138-79 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 18:42:46.238032 ip-10-0-138-79 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 36120ae2d9ef435297815be5c2cab028 -- Sep 16 18:44:22.235834 ip-10-0-138-79 systemd[1]: Starting Kubernetes Kubelet... Sep 16 18:44:22.657787 ip-10-0-138-79 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:44:22.657787 ip-10-0-138-79 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 18:44:22.657787 ip-10-0-138-79 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:44:22.657787 ip-10-0-138-79 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 18:44:22.657787 ip-10-0-138-79 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:44:22.660564 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.660440 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 18:44:22.667769 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667741 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:22.667769 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667761 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:22.667769 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667765 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:22.667769 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667768 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:22.667769 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667773 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:22.667769 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667776 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667779 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667782 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667784 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667789 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667794 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667797 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667800 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667803 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667805 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667808 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667810 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667813 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667816 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667818 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667821 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667823 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667826 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667828 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:22.667987 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667831 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667837 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667840 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667842 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667845 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667847 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667850 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667852 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667855 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667857 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667860 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667862 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667865 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667867 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667870 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667873 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667875 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667878 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667882 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667886 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:22.668444 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667889 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667892 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667894 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667897 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667899 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667902 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667904 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667907 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667910 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667912 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667917 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667921 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667924 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667927 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667930 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667933 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667936 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667939 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667941 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:22.668989 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667944 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667947 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667949 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667952 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667955 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667957 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667960 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667963 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667966 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667969 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667972 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667975 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667977 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667980 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667983 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667985 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667988 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667990 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667995 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.667997 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:22.669463 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668001 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668004 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668007 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668009 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668012 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668014 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668017 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668019 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668021 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668024 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668026 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668028 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668031 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668033 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668036 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.668038 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669828 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669837 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669840 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669843 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:22.669965 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669846 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669849 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669852 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669855 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669858 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669860 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669863 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669865 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669868 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669870 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669872 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669875 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669877 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669880 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669884 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669887 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669889 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669892 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669895 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669897 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:22.670441 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669900 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669902 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669905 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669908 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669910 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669913 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669915 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669918 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669920 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669922 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669925 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669928 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669930 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669932 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669935 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669937 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669940 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669942 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669945 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:22.670953 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669947 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669949 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669952 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669954 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669957 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669959 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669962 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669966 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669969 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669971 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669974 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669977 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669979 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669982 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669984 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669988 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669991 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669993 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669995 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.669998 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:22.671459 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670001 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670003 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670006 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670008 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670011 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670013 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670016 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670019 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670021 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670024 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670026 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670029 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670033 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670036 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670039 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670042 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670045 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670048 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670050 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670055 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:22.671966 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670058 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670061 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670065 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670068 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670072 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670074 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670077 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670080 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670082 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670084 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670087 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670089 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670092 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670094 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670097 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670100 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670476 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670495 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670499 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:22.672461 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670502 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670504 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670507 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670509 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670514 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670518 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670521 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670524 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670526 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670529 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670532 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670534 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670540 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670544 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670546 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670549 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670551 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670554 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670556 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670559 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:22.672934 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670562 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670564 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670566 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670569 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670571 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670574 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670576 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670578 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670582 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670585 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670588 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670591 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670594 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670596 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670599 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670602 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670605 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670608 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670610 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:22.673424 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670612 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670615 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670617 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670620 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670622 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670625 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670627 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670631 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670633 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670636 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670638 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670642 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670645 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670647 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670650 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670652 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670655 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670657 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670660 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670662 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:22.673933 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670665 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670667 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670670 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670673 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670675 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670678 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670680 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670683 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670686 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670688 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670690 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670693 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670695 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670697 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670700 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670703 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670705 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670707 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670710 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670712 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:22.674430 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670715 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670717 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670720 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670722 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670725 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670727 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670729 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670731 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670734 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670736 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670738 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670741 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670743 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670746 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670748 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670751 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.670753 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670829 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670873 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670880 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 18:44:22.675032 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670884 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670888 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670891 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670895 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670900 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670903 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670905 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670909 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670912 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670916 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670919 2576 flags.go:64] FLAG: --cgroup-root="" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670921 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670924 2576 flags.go:64] FLAG: --client-ca-file="" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670928 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670931 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670936 2576 flags.go:64] FLAG: --cluster-domain="" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670939 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670942 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670945 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670948 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670952 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670955 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670958 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 18:44:22.675542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670961 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670964 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670973 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670980 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670983 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670986 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670991 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670994 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670997 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.670999 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671002 2576 flags.go:64] FLAG: --enable-server="true" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671005 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671009 2576 flags.go:64] FLAG: --event-burst="100" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671012 2576 flags.go:64] FLAG: --event-qps="50" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671015 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671018 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671021 2576 flags.go:64] FLAG: --eviction-hard="" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671025 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671028 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671031 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671034 2576 flags.go:64] FLAG: --eviction-soft="" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671037 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671040 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671043 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671046 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 18:44:22.676121 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671049 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671051 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671054 2576 flags.go:64] FLAG: --feature-gates="" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671058 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671061 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671064 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671067 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671070 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671073 2576 flags.go:64] FLAG: --help="false" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671075 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-138-79.ec2.internal" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671080 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671082 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671086 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671089 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671093 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671095 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671098 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671101 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671104 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671106 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671110 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671112 2576 flags.go:64] FLAG: --kube-reserved="" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671115 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671118 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 18:44:22.676790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671121 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671123 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671126 2576 flags.go:64] FLAG: --lock-file="" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671129 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671132 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671134 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671140 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671143 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671146 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671149 2576 flags.go:64] FLAG: --logging-format="text" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671152 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671155 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671158 2576 flags.go:64] FLAG: --manifest-url="" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671160 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671165 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671167 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671171 2576 flags.go:64] FLAG: --max-pods="110" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671174 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671179 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671181 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671185 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671187 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671190 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671193 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671200 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 18:44:22.677370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671203 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671206 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671209 2576 flags.go:64] FLAG: --pod-cidr="" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671212 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671217 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671220 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671222 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671227 2576 flags.go:64] FLAG: --port="10250" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671230 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671232 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cc0c69c21309d203" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671236 2576 flags.go:64] FLAG: --qos-reserved="" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671238 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671241 2576 flags.go:64] FLAG: --register-node="true" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671244 2576 flags.go:64] FLAG: --register-with-taints="" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671248 2576 flags.go:64] FLAG: --registry-burst="10" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671251 2576 flags.go:64] FLAG: --registry-qps="5" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671254 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671257 2576 flags.go:64] FLAG: --reserved-memory="" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671260 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671263 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671266 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671269 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671272 2576 flags.go:64] FLAG: --runonce="false" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671274 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671277 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 18:44:22.678000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671281 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671284 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671287 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671289 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671292 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671295 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671298 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671301 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671303 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671306 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671309 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671312 2576 flags.go:64] FLAG: --system-cgroups="" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671315 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671320 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671325 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671328 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671333 2576 flags.go:64] FLAG: --tls-min-version="" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671336 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671338 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671341 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671344 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671347 2576 flags.go:64] FLAG: --v="2" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671351 2576 flags.go:64] FLAG: --version="false" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671355 2576 flags.go:64] FLAG: --vmodule="" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671359 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 18:44:22.678655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.671362 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671474 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671478 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671495 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671498 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671501 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671504 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671508 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671511 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671514 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671516 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671519 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671522 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671525 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671527 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671530 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671532 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671535 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671537 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671540 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:22.679734 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671542 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671546 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671548 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671551 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671553 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671556 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671558 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671561 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671563 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671566 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671568 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671571 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671574 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671576 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671578 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671581 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671583 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671586 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671588 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671592 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:22.680584 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671595 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671597 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671600 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671604 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671608 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671611 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671615 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671618 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671621 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671624 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671642 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671646 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671649 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671655 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671658 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671661 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671664 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671666 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671669 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:22.681223 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671672 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671674 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671677 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671679 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671681 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671686 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671689 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671692 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671695 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671697 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671700 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671703 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671706 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671709 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671711 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671714 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671716 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671719 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671722 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:22.681811 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671724 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671726 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671729 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671732 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671734 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671737 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671739 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671743 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671745 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671748 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671750 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671753 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671755 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671758 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671760 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671763 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671765 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671768 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671770 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671773 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:22.682661 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671775 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.671778 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.672465 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.680867 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.680889 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.681777 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.681795 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681921 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681930 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681935 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681940 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681944 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681948 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681953 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:22.683421 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681958 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681962 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681966 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681971 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681975 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681979 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681984 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681988 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681992 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.681996 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682000 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682005 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682010 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682014 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682018 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682022 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682027 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682030 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682034 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:22.683793 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682038 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682045 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682051 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682058 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682075 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682080 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682085 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682089 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682094 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682099 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682104 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682108 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682112 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682116 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682121 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682124 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682128 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682132 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682137 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682141 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:22.684546 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682145 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682149 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682153 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682158 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682162 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682166 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682170 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682174 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682178 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682183 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682187 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682191 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682195 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682199 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682203 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682207 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682212 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682216 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682220 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682224 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:22.685378 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682228 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682232 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682236 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682240 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682244 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682248 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682252 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682256 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682261 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682265 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682269 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682274 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682278 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682282 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682288 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682294 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682299 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682303 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682308 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682313 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:22.685900 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682318 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682322 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682326 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682330 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682336 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682342 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682347 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682352 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682356 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682361 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682374 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682378 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682382 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.682390 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:44:22.686620 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682591 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682601 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682607 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682612 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682616 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682620 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682625 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682629 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682633 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682637 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682641 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682645 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682649 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682653 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682657 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682661 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682665 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682669 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682673 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:44:22.686975 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682676 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682680 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682684 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682688 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682692 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682696 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682701 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682705 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682709 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682714 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682718 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682723 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682727 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682731 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682735 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682739 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682743 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682747 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682754 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:44:22.687510 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682760 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682764 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682769 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682774 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682779 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682783 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682787 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682791 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682796 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682799 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682804 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682808 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682812 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682818 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682824 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682828 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682832 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682836 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682840 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682844 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:44:22.687977 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682848 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682852 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682856 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682861 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682866 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682870 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682874 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682878 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682882 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682886 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682890 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682894 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682898 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682902 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682906 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682910 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682914 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682918 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682922 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682926 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:44:22.688626 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682930 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682934 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682939 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682942 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682947 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682951 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682955 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682959 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682964 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682967 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682971 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682975 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682980 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682984 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682988 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682992 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.682997 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.683001 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.683005 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.683009 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:44:22.689136 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:22.683012 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:44:22.689658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.683021 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:44:22.689658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.683167 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 18:44:22.689658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.686057 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 18:44:22.689658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.687227 2576 server.go:1019] "Starting client certificate rotation" Sep 16 18:44:22.689658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.687335 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:44:22.689658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.687370 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:44:22.690211 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.690195 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:44:22.693401 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.693382 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:44:22.706395 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.706235 2576 log.go:25] "Validated CRI v1 runtime API" Sep 16 18:44:22.710072 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.710055 2576 log.go:25] "Validated CRI v1 image API" Sep 16 18:44:22.710942 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.710925 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 18:44:22.715128 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.715106 2576 fs.go:135] Filesystem UUIDs: map[222de99b-1105-43a3-9034-88a9d21a046f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e3296ed9-c307-4b48-b0f7-f489805f2a6a:/dev/nvme0n1p3] Sep 16 18:44:22.715201 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.715126 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 18:44:22.720330 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.720212 2576 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:22.718824706 +0000 UTC m=+0.364919502 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086822 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec239ab75a92bdf3e0b4fc35975195e8 SystemUUID:ec239ab7-5a92-bdf3-e0b4-fc35975195e8 BootID:36120ae2-d9ef-4352-9781-5be5c2cab028 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:2e:41:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:2e:41:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:44:e3:32:be:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 18:44:22.720330 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.720318 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 18:44:22.720511 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.720421 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 18:44:22.721818 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.721801 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:44:22.724757 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.724728 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 18:44:22.724921 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.724763 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 18:44:22.725656 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.725644 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 18:44:22.725718 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.725662 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 18:44:22.725718 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.725672 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:44:22.726450 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.726438 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:44:22.728050 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.728038 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 18:44:22.731216 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731202 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:44:22.731345 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731334 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 18:44:22.731413 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731373 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 16 18:44:22.731413 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731388 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 18:44:22.731413 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731409 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 18:44:22.731569 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731422 2576 kubelet.go:398] "Adding apiserver pod source" Sep 16 18:44:22.731569 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731434 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 18:44:22.731894 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731880 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:44:22.731956 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.731904 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:44:22.734328 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.734313 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 18:44:22.736432 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.736417 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 18:44:22.736507 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.736447 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 18:44:22.738406 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738394 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 18:44:22.738458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738411 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 18:44:22.738458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738417 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 18:44:22.738458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738422 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 18:44:22.738458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738435 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 18:44:22.738458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738444 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 18:44:22.738458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738460 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 18:44:22.738640 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738465 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 18:44:22.738640 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738472 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 18:44:22.738640 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738476 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 18:44:22.738640 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738510 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 18:44:22.738640 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.738516 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 18:44:22.740603 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.740583 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 18:44:22.740727 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.740717 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 18:44:22.742855 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.742830 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 18:44:22.742936 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.742856 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 18:44:22.742936 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.742870 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 18:44:22.745104 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.745089 2576 server.go:1268] "Started kubelet" Sep 16 18:44:22.745166 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.745146 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 18:44:22.745251 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.745221 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 18:44:22.745286 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.745274 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 18:44:22.745834 ip-10-0-138-79 systemd[1]: Started Kubernetes Kubelet. Sep 16 18:44:22.748049 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.748031 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 16 18:44:22.749248 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.749226 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 18:44:22.750007 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.749989 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 18:44:22.757287 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.757272 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 18:44:22.758096 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758080 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 18:44:22.758188 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.758170 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:22.758835 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758817 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 18:44:22.758835 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758819 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 18:44:22.758835 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758839 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 18:44:22.759054 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758844 2576 factory.go:55] Registering systemd factory Sep 16 18:44:22.759054 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758859 2576 factory.go:223] Registration of the systemd container factory successfully Sep 16 18:44:22.759054 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758902 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 16 18:44:22.759054 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.758910 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 16 18:44:22.759186 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.759073 2576 factory.go:153] Registering CRI-O factory Sep 16 18:44:22.759186 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.759083 2576 factory.go:223] Registration of the crio container factory successfully Sep 16 18:44:22.759186 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.759139 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 18:44:22.759186 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.759155 2576 factory.go:103] Registering Raw factory Sep 16 18:44:22.759186 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.759167 2576 manager.go:1196] Started watching for new ooms in manager Sep 16 18:44:22.760129 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.760115 2576 manager.go:349] Starting recovery of all containers Sep 16 18:44:22.760415 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.759252 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-79.ec2.internal.18d5e167c48c414e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-79.ec2.internal,UID:ip-10-0-138-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-79.ec2.internal,},FirstTimestamp:2026-09-16 18:44:22.745047374 +0000 UTC m=+0.391142174,LastTimestamp:2026-09-16 18:44:22.745047374 +0000 UTC m=+0.391142174,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-79.ec2.internal,}" Sep 16 18:44:22.771324 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.771139 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 18:44:22.771324 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.771277 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 18:44:22.772801 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.771558 2576 manager.go:354] Recovery completed Sep 16 18:44:22.775039 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.775019 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qz6ck" Sep 16 18:44:22.777064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.777051 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:22.778710 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.778692 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:22.778772 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.778724 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:22.778772 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.778739 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:22.779249 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.779236 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 18:44:22.779249 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.779248 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 18:44:22.779332 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.779265 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:44:22.779332 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.779267 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:22.779936 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.779920 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:22.780002 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.779992 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:22.780059 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.780009 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:22.780551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.780538 2576 policy_none.go:49] "None policy: Start" Sep 16 18:44:22.780597 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.780556 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 18:44:22.780597 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.780566 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 18:44:22.781034 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.780969 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-79.ec2.internal.18d5e167c68de5bc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-79.ec2.internal,UID:ip-10-0-138-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-79.ec2.internal,},FirstTimestamp:2026-09-16 18:44:22.778709436 +0000 UTC m=+0.424804236,LastTimestamp:2026-09-16 18:44:22.778709436 +0000 UTC m=+0.424804236,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-79.ec2.internal,}" Sep 16 18:44:22.781658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.781646 2576 policy_none.go:47] "Start" Sep 16 18:44:22.783333 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.783318 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qz6ck" Sep 16 18:44:22.824104 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.824082 2576 manager.go:337] "Starting Device Plugin manager" Sep 16 18:44:22.824237 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.824131 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 18:44:22.824237 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.824144 2576 server.go:87] "Starting device plugin registration server" Sep 16 18:44:22.824431 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.824422 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 18:44:22.824552 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.824435 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 18:44:22.824843 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.824829 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 18:44:22.824932 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.824902 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 18:44:22.824932 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.824910 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 18:44:22.825751 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.825551 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 18:44:22.825751 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.825602 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:22.875655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.875618 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 18:44:22.877052 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.877032 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 18:44:22.877106 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.877089 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 18:44:22.877139 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.877114 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 18:44:22.877184 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.877155 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 18:44:22.879755 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.879734 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:22.924990 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.924962 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:22.925769 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.925754 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:22.925834 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.925785 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:22.925834 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.925795 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:22.925834 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.925819 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-79.ec2.internal" Sep 16 18:44:22.940111 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.940026 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-79.ec2.internal" Sep 16 18:44:22.940111 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.940051 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-79.ec2.internal\": node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:22.967942 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:22.967919 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:22.977999 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.977979 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal"] Sep 16 18:44:22.978053 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.978040 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:22.979383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.979367 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:22.979464 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.979401 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:22.979464 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.979426 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:22.980643 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.980628 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:22.980776 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.980760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:22.980825 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.980796 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:22.981302 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.981289 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:22.981359 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.981314 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:22.981359 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.981323 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:22.981421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.981293 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:22.981421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.981389 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:22.981421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.981399 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:22.982572 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.982553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" Sep 16 18:44:22.982655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.982580 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:44:22.983218 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.983203 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:44:22.983283 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.983230 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:44:22.983283 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:22.983240 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:44:23.008008 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.007989 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-79.ec2.internal\" not found" node="ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.012167 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.012151 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-79.ec2.internal\" not found" node="ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.060875 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.060850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8291ccbc2b9783dfbef0924ad26b3f7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal\" (UID: \"8291ccbc2b9783dfbef0924ad26b3f7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.060875 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.060876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8291ccbc2b9783dfbef0924ad26b3f7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal\" (UID: \"8291ccbc2b9783dfbef0924ad26b3f7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.061028 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.060893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/250437f78067fb73286aae1b34e08bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-138-79.ec2.internal\" (UID: \"250437f78067fb73286aae1b34e08bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.068977 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.068943 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.161684 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.161655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8291ccbc2b9783dfbef0924ad26b3f7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal\" (UID: \"8291ccbc2b9783dfbef0924ad26b3f7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.161684 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.161683 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8291ccbc2b9783dfbef0924ad26b3f7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal\" (UID: \"8291ccbc2b9783dfbef0924ad26b3f7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.161840 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.161705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/250437f78067fb73286aae1b34e08bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-138-79.ec2.internal\" (UID: \"250437f78067fb73286aae1b34e08bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.161840 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.161747 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8291ccbc2b9783dfbef0924ad26b3f7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal\" (UID: \"8291ccbc2b9783dfbef0924ad26b3f7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.161840 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.161765 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/250437f78067fb73286aae1b34e08bfc-config\") pod \"kube-apiserver-proxy-ip-10-0-138-79.ec2.internal\" (UID: \"250437f78067fb73286aae1b34e08bfc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.161840 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.161747 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8291ccbc2b9783dfbef0924ad26b3f7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal\" (UID: \"8291ccbc2b9783dfbef0924ad26b3f7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.169765 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.169730 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.270521 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.270462 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.310658 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.310623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.315267 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.315253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" Sep 16 18:44:23.371198 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.371166 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.471665 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.471639 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.572224 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.572152 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.672652 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.672621 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.686786 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.686762 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 18:44:23.686931 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.686913 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:44:23.773697 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.773669 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.785855 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.785816 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:22 +0000 UTC" deadline="2028-06-22 03:44:08.943953294 +0000 UTC" Sep 16 18:44:23.785855 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.785846 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15464h59m45.158110773s" Sep 16 18:44:23.818881 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:23.818851 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8291ccbc2b9783dfbef0924ad26b3f7f.slice/crio-77c1c8d6f578c0d7d9aacc19c58ac262c726031da6c98be74fe713448313cd77 WatchSource:0}: Error finding container 77c1c8d6f578c0d7d9aacc19c58ac262c726031da6c98be74fe713448313cd77: Status 404 returned error can't find the container with id 77c1c8d6f578c0d7d9aacc19c58ac262c726031da6c98be74fe713448313cd77 Sep 16 18:44:23.819376 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:23.819362 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250437f78067fb73286aae1b34e08bfc.slice/crio-b1203268f2ab7bfaadcfd5566c4e720e1dcfd12d1deaa3ab591be90a3e164d7b WatchSource:0}: Error finding container b1203268f2ab7bfaadcfd5566c4e720e1dcfd12d1deaa3ab591be90a3e164d7b: Status 404 returned error can't find the container with id b1203268f2ab7bfaadcfd5566c4e720e1dcfd12d1deaa3ab591be90a3e164d7b Sep 16 18:44:23.824542 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.824528 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:44:23.874326 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.874298 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:23.880939 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.880903 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" event={"ID":"250437f78067fb73286aae1b34e08bfc","Type":"ContainerStarted","Data":"b1203268f2ab7bfaadcfd5566c4e720e1dcfd12d1deaa3ab591be90a3e164d7b"} Sep 16 18:44:23.881997 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.881979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" event={"ID":"8291ccbc2b9783dfbef0924ad26b3f7f","Type":"ContainerStarted","Data":"77c1c8d6f578c0d7d9aacc19c58ac262c726031da6c98be74fe713448313cd77"} Sep 16 18:44:23.912349 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:23.912326 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:23.974428 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:23.974401 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:24.020449 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.020421 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:24.075176 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.075107 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:24.175332 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.175296 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:24.275764 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.275717 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-79.ec2.internal\" not found" Sep 16 18:44:24.309159 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.309124 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:24.359583 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.359247 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" Sep 16 18:44:24.375126 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.375094 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:44:24.376247 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.376220 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" Sep 16 18:44:24.385880 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.385856 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:44:24.713452 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.713186 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:44:24.732664 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.732441 2576 apiserver.go:52] "Watching apiserver" Sep 16 18:44:24.741027 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.740146 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 18:44:24.741027 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.740628 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4","openshift-cluster-node-tuning-operator/tuned-p72gp","openshift-image-registry/node-ca-rcnm2","openshift-multus/network-metrics-daemon-b47hq","openshift-ovn-kubernetes/ovnkube-node-xx2wj","kube-system/global-pull-secret-syncer-sm66h","kube-system/konnectivity-agent-htfgb","openshift-dns/node-resolver-q4p8g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal","openshift-multus/multus-additional-cni-plugins-ldjt8","openshift-multus/multus-dwbzs","openshift-network-diagnostics/network-check-target-w8b6s","openshift-network-operator/iptables-alerter-h9qbx","kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal"] Sep 16 18:44:24.742386 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.742354 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:24.743712 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.743688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.744915 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.744897 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6z7d\"" Sep 16 18:44:24.745029 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.744949 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 18:44:24.745029 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.745023 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 18:44:24.745672 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.745221 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.746001 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.745969 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:44:24.746399 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.746378 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 18:44:24.746906 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.746644 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:24.746906 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.746748 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:24.748168 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.747841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.748892 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.748867 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 18:44:24.748996 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.748895 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mr2vk\"" Sep 16 18:44:24.749150 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.749128 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 18:44:24.749150 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.749147 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vgkvj\"" Sep 16 18:44:24.749311 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.749170 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 18:44:24.749311 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.749128 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 18:44:24.749412 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.749395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.749483 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.749462 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:24.750222 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.750071 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 18:44:24.750222 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.750175 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 18:44:24.750782 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.750761 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 18:44:24.751056 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.751040 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.751624 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.751603 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 18:44:24.751744 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.751699 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r75fm\"" Sep 16 18:44:24.751873 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.751855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 18:44:24.752095 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.752076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 18:44:24.752633 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.752603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.753803 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.753779 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 18:44:24.753956 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.753832 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mvcs8\"" Sep 16 18:44:24.753956 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.753951 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 18:44:24.754260 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.754240 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.755318 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.755298 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 18:44:24.755419 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.755334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x7tp8\"" Sep 16 18:44:24.755628 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.755605 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 18:44:24.755769 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.755742 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 18:44:24.756423 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.756323 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.760685 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.760633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:24.760780 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.760761 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:24.761079 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.761055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.761452 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.761429 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 18:44:24.763425 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.763406 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:44:24.764314 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.764292 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-46hks\"" Sep 16 18:44:24.764578 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.764525 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 18:44:24.765037 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.764977 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 18:44:24.765967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.765923 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 18:44:24.766818 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.766798 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lx79v\"" Sep 16 18:44:24.767047 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.767028 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 18:44:24.767613 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.767385 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 18:44:24.767613 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.767477 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 18:44:24.767613 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.767480 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 18:44:24.768227 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.768209 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rfnnz\"" Sep 16 18:44:24.768455 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.768438 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 18:44:24.770800 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysconfig\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.770947 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770806 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-host\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.770947 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-socket-dir-parent\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.770947 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-hostroot\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.770947 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-socket-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.770947 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-etc-selinux\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.770947 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vmb\" (UniqueName: \"kubernetes.io/projected/d1add103-27f0-42c2-b4c2-0b96269754fe-kube-api-access-k5vmb\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.770987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-cni-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65fdr\" (UniqueName: \"kubernetes.io/projected/e3fd2611-8884-4a84-861e-1342cee39e99-kube-api-access-65fdr\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771037 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b647a36-5a97-423e-af52-e5cf39fa9a1d-konnectivity-ca\") pod \"konnectivity-agent-htfgb\" (UID: \"4b647a36-5a97-423e-af52-e5cf39fa9a1d\") " pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-system-cni-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771084 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-systemd\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-cnibin\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-k8s-cni-cncf-io\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-cni-bin\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-env-overrides\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b647a36-5a97-423e-af52-e5cf39fa9a1d-agent-certs\") pod \"konnectivity-agent-htfgb\" (UID: \"4b647a36-5a97-423e-af52-e5cf39fa9a1d\") " pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-cni-netd\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.771273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovnkube-config\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-sys-fs\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-run\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23954e0d-2798-4c79-8b43-244302141d00-host\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52c9w\" (UniqueName: \"kubernetes.io/projected/321dd925-57a6-48a7-b8b4-da2ad9e064cd-kube-api-access-52c9w\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80653a37-6288-47de-8d79-e913d7a23687-cni-binary-copy\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovnkube-script-lib\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysctl-conf\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-system-cni-dir\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45sb\" (UniqueName: \"kubernetes.io/projected/80653a37-6288-47de-8d79-e913d7a23687-kube-api-access-c45sb\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1add103-27f0-42c2-b4c2-0b96269754fe-iptables-alerter-script\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-cnibin\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-kubelet\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-slash\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-var-lib-kubelet\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771725 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-tuned\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.772113 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771824 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-ovn\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f722\" (UniqueName: \"kubernetes.io/projected/c6d71861-fc83-4426-9f3e-15ca25b91f00-kube-api-access-2f722\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8dc\" (UniqueName: \"kubernetes.io/projected/23954e0d-2798-4c79-8b43-244302141d00-kube-api-access-mm8dc\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.771949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-cni-multus\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772035 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-log-socket\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-cni-bin\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-kubelet\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-systemd\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlr5\" (UniqueName: \"kubernetes.io/projected/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-kube-api-access-mhlr5\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772141 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dkk8\" (UniqueName: \"kubernetes.io/projected/59781a33-3f1e-46cc-864e-a57b0e76fc67-kube-api-access-4dkk8\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-etc-kubernetes\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772190 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-lib-modules\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772204 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/321dd925-57a6-48a7-b8b4-da2ad9e064cd-tmp\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.772850 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772219 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2e9015b-775b-4a7d-8542-9a1222e7f67b-kubelet-config\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-tmp-dir\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1add103-27f0-42c2-b4c2-0b96269754fe-host-slash\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-modprobe-d\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-netns\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-multus-certs\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-var-lib-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2e9015b-775b-4a7d-8542-9a1222e7f67b-dbus\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-sys\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-registration-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-device-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysctl-d\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:24.773467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-run-netns\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-hosts-file\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23954e0d-2798-4c79-8b43-244302141d00-serviceca\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-kubernetes\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n982l\" (UniqueName: \"kubernetes.io/projected/5f973677-db95-4065-851a-c26414a4c960-kube-api-access-n982l\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772746 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80653a37-6288-47de-8d79-e913d7a23687-multus-daemon-config\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovn-node-metrics-cert\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-conf-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-systemd-units\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-etc-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.774035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.772848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-node-log\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.781053 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.781027 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:44:24.836753 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.836727 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mrk5w" Sep 16 18:44:24.860698 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.860666 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mrk5w" Sep 16 18:44:24.873568 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.873540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23954e0d-2798-4c79-8b43-244302141d00-serviceca\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.873705 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.873578 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-kubernetes\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.873705 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.873596 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n982l\" (UniqueName: \"kubernetes.io/projected/5f973677-db95-4065-851a-c26414a4c960-kube-api-access-n982l\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.873833 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.873717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80653a37-6288-47de-8d79-e913d7a23687-multus-daemon-config\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.873968 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.873937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovn-node-metrics-cert\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.873968 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.873956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-kubernetes\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.874101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.873984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.874101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-conf-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.874101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-systemd-units\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.874101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874061 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-etc-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.874101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874071 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23954e0d-2798-4c79-8b43-244302141d00-serviceca\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.874101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-node-log\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysconfig\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-host\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874167 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-systemd-units\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-socket-dir-parent\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874209 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-hostroot\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-socket-dir-parent\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-socket-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-etc-selinux\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-etc-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874286 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vmb\" (UniqueName: \"kubernetes.io/projected/d1add103-27f0-42c2-b4c2-0b96269754fe-kube-api-access-k5vmb\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874307 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-node-log\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-cni-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65fdr\" (UniqueName: \"kubernetes.io/projected/e3fd2611-8884-4a84-861e-1342cee39e99-kube-api-access-65fdr\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b647a36-5a97-423e-af52-e5cf39fa9a1d-konnectivity-ca\") pod \"konnectivity-agent-htfgb\" (UID: \"4b647a36-5a97-423e-af52-e5cf39fa9a1d\") " pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:24.874414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874418 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-host\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-system-cni-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/80653a37-6288-47de-8d79-e913d7a23687-multus-daemon-config\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-systemd\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-cnibin\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874487 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-k8s-cni-cncf-io\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874766 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-k8s-cni-cncf-io\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysconfig\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874799 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-systemd\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-hostroot\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-system-cni-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-cni-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-etc-selinux\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874636 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874910 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-cnibin\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-cni-bin\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-cni-bin\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.875276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-socket-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.874982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-env-overrides\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b647a36-5a97-423e-af52-e5cf39fa9a1d-agent-certs\") pod \"konnectivity-agent-htfgb\" (UID: \"4b647a36-5a97-423e-af52-e5cf39fa9a1d\") " pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-multus-conf-dir\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-cni-netd\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovnkube-config\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4b647a36-5a97-423e-af52-e5cf39fa9a1d-konnectivity-ca\") pod \"konnectivity-agent-htfgb\" (UID: \"4b647a36-5a97-423e-af52-e5cf39fa9a1d\") " pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-sys-fs\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-run-ovn-kubernetes\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-run\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-cni-netd\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23954e0d-2798-4c79-8b43-244302141d00-host\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-run\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52c9w\" (UniqueName: \"kubernetes.io/projected/321dd925-57a6-48a7-b8b4-da2ad9e064cd-kube-api-access-52c9w\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80653a37-6288-47de-8d79-e913d7a23687-cni-binary-copy\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875249 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-sys-fs\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.876203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovnkube-script-lib\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysctl-conf\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-system-cni-dir\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45sb\" (UniqueName: \"kubernetes.io/projected/80653a37-6288-47de-8d79-e913d7a23687-kube-api-access-c45sb\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1add103-27f0-42c2-b4c2-0b96269754fe-iptables-alerter-script\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-cnibin\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-kubelet\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-slash\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-var-lib-kubelet\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875494 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23954e0d-2798-4c79-8b43-244302141d00-host\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-tuned\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovnkube-config\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-ovn\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f722\" (UniqueName: \"kubernetes.io/projected/c6d71861-fc83-4426-9f3e-15ca25b91f00-kube-api-access-2f722\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8dc\" (UniqueName: \"kubernetes.io/projected/23954e0d-2798-4c79-8b43-244302141d00-kube-api-access-mm8dc\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:24.877188 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-cni-multus\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-system-cni-dir\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875707 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysctl-conf\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-log-socket\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875768 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-cni-bin\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875818 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-kubelet\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-systemd\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlr5\" (UniqueName: \"kubernetes.io/projected/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-kube-api-access-mhlr5\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875889 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dkk8\" (UniqueName: \"kubernetes.io/projected/59781a33-3f1e-46cc-864e-a57b0e76fc67-kube-api-access-4dkk8\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-etc-kubernetes\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-lib-modules\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/321dd925-57a6-48a7-b8b4-da2ad9e064cd-tmp\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2e9015b-775b-4a7d-8542-9a1222e7f67b-kubelet-config\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875973 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-kubelet\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-tmp-dir\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876025 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-log-socket\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.878064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1add103-27f0-42c2-b4c2-0b96269754fe-host-slash\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876052 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovnkube-script-lib\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876066 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-cni-bin\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d1add103-27f0-42c2-b4c2-0b96269754fe-host-slash\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-modprobe-d\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-systemd\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-netns\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876163 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-multus-certs\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-var-lib-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2e9015b-775b-4a7d-8542-9a1222e7f67b-dbus\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876252 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d1add103-27f0-42c2-b4c2-0b96269754fe-iptables-alerter-script\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-sys\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-registration-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-device-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysctl-d\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.879020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-etc-kubernetes\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-run-netns\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-lib-modules\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876598 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-slash\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-hosts-file\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876642 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-cnibin\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876702 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-hosts-file\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876774 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-kubelet\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.875459 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c6d71861-fc83-4426-9f3e-15ca25b91f00-env-overrides\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876820 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-netns\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-run-multus-certs\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876790 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-modprobe-d\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876900 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-var-lib-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.879922 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876918 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/80653a37-6288-47de-8d79-e913d7a23687-cni-binary-copy\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876951 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-registration-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.876953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59781a33-3f1e-46cc-864e-a57b0e76fc67-device-dir\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877007 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2e9015b-775b-4a7d-8542-9a1222e7f67b-dbus\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-sysctl-d\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.877094 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.877182 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret podName:f2e9015b-775b-4a7d-8542-9a1222e7f67b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.377144351 +0000 UTC m=+3.023239153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret") pod "global-pull-secret-syncer-sm66h" (UID: "f2e9015b-775b-4a7d-8542-9a1222e7f67b") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877199 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5f973677-db95-4065-851a-c26414a4c960-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877428 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-sys\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-run-netns\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5f973677-db95-4065-851a-c26414a4c960-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877486 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-openvswitch\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877493 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/321dd925-57a6-48a7-b8b4-da2ad9e064cd-var-lib-kubelet\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877576 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2e9015b-775b-4a7d-8542-9a1222e7f67b-kubelet-config\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c6d71861-fc83-4426-9f3e-15ca25b91f00-run-ovn\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877645 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/80653a37-6288-47de-8d79-e913d7a23687-host-var-lib-cni-multus\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.880667 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.877764 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:24.881381 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.877817 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.377799447 +0000 UTC m=+3.023894235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:24.881381 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.877919 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-tmp-dir\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.881381 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.879323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/321dd925-57a6-48a7-b8b4-da2ad9e064cd-etc-tuned\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.881381 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.879354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/321dd925-57a6-48a7-b8b4-da2ad9e064cd-tmp\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.881381 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.879435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c6d71861-fc83-4426-9f3e-15ca25b91f00-ovn-node-metrics-cert\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.881381 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.879527 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4b647a36-5a97-423e-af52-e5cf39fa9a1d-agent-certs\") pod \"konnectivity-agent-htfgb\" (UID: \"4b647a36-5a97-423e-af52-e5cf39fa9a1d\") " pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:24.891755 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.891733 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:24.891891 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.891759 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:24.891891 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.891774 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dtdlx for pod openshift-network-diagnostics/network-check-target-w8b6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:24.891891 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:24.891836 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx podName:6b6ef13b-8221-4c89-981c-d17ed779bbca nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.391818887 +0000 UTC m=+3.037913696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dtdlx" (UniqueName: "kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx") pod "network-check-target-w8b6s" (UID: "6b6ef13b-8221-4c89-981c-d17ed779bbca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:24.897580 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.893980 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n982l\" (UniqueName: \"kubernetes.io/projected/5f973677-db95-4065-851a-c26414a4c960-kube-api-access-n982l\") pod \"multus-additional-cni-plugins-ldjt8\" (UID: \"5f973677-db95-4065-851a-c26414a4c960\") " pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:24.897580 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.894664 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlr5\" (UniqueName: \"kubernetes.io/projected/2e46d10a-d3c7-4b28-bab1-a16ffbe596b1-kube-api-access-mhlr5\") pod \"node-resolver-q4p8g\" (UID: \"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1\") " pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:24.897580 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.895248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vmb\" (UniqueName: \"kubernetes.io/projected/d1add103-27f0-42c2-b4c2-0b96269754fe-kube-api-access-k5vmb\") pod \"iptables-alerter-h9qbx\" (UID: \"d1add103-27f0-42c2-b4c2-0b96269754fe\") " pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:24.897580 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.896132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52c9w\" (UniqueName: \"kubernetes.io/projected/321dd925-57a6-48a7-b8b4-da2ad9e064cd-kube-api-access-52c9w\") pod \"tuned-p72gp\" (UID: \"321dd925-57a6-48a7-b8b4-da2ad9e064cd\") " pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:24.897580 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.896814 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dkk8\" (UniqueName: \"kubernetes.io/projected/59781a33-3f1e-46cc-864e-a57b0e76fc67-kube-api-access-4dkk8\") pod \"aws-ebs-csi-driver-node-jhkv4\" (UID: \"59781a33-3f1e-46cc-864e-a57b0e76fc67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:24.897898 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.897778 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45sb\" (UniqueName: \"kubernetes.io/projected/80653a37-6288-47de-8d79-e913d7a23687-kube-api-access-c45sb\") pod \"multus-dwbzs\" (UID: \"80653a37-6288-47de-8d79-e913d7a23687\") " pod="openshift-multus/multus-dwbzs" Sep 16 18:44:24.898023 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.898005 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f722\" (UniqueName: \"kubernetes.io/projected/c6d71861-fc83-4426-9f3e-15ca25b91f00-kube-api-access-2f722\") pod \"ovnkube-node-xx2wj\" (UID: \"c6d71861-fc83-4426-9f3e-15ca25b91f00\") " pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:24.899192 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.899169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65fdr\" (UniqueName: \"kubernetes.io/projected/e3fd2611-8884-4a84-861e-1342cee39e99-kube-api-access-65fdr\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:24.899934 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:24.899914 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8dc\" (UniqueName: \"kubernetes.io/projected/23954e0d-2798-4c79-8b43-244302141d00-kube-api-access-mm8dc\") pod \"node-ca-rcnm2\" (UID: \"23954e0d-2798-4c79-8b43-244302141d00\") " pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:25.061948 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.061846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:25.070247 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.070219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-h9qbx" Sep 16 18:44:25.081415 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.081387 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dwbzs" Sep 16 18:44:25.088294 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.088269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rcnm2" Sep 16 18:44:25.096396 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.096365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:25.103170 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.103152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4p8g" Sep 16 18:44:25.109381 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.109363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" Sep 16 18:44:25.116423 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.116404 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p72gp" Sep 16 18:44:25.121116 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.121094 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" Sep 16 18:44:25.381034 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.380944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:25.381034 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.381001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:25.381237 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.381105 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:25.381237 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.381127 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:25.381237 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.381156 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret podName:f2e9015b-775b-4a7d-8542-9a1222e7f67b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.381142196 +0000 UTC m=+4.027236995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret") pod "global-pull-secret-syncer-sm66h" (UID: "f2e9015b-775b-4a7d-8542-9a1222e7f67b") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:25.381237 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.381189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.381173564 +0000 UTC m=+4.027268371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:25.481612 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.481570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:25.481785 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.481742 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:25.481785 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.481767 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:25.481785 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.481778 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dtdlx for pod openshift-network-diagnostics/network-check-target-w8b6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:25.481926 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.481847 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx podName:6b6ef13b-8221-4c89-981c-d17ed779bbca nodeName:}" failed. No retries permitted until 2026-09-16 18:44:26.481830362 +0000 UTC m=+4.127925148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dtdlx" (UniqueName: "kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx") pod "network-check-target-w8b6s" (UID: "6b6ef13b-8221-4c89-981c-d17ed779bbca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:25.831666 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:25.831637 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6d71861_fc83_4426_9f3e_15ca25b91f00.slice/crio-cc4650fd74b7ed3d51e0cd63bd4981fff13a6a5fceabd61bd09420c3138848c2 WatchSource:0}: Error finding container cc4650fd74b7ed3d51e0cd63bd4981fff13a6a5fceabd61bd09420c3138848c2: Status 404 returned error can't find the container with id cc4650fd74b7ed3d51e0cd63bd4981fff13a6a5fceabd61bd09420c3138848c2 Sep 16 18:44:25.832326 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:25.832298 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321dd925_57a6_48a7_b8b4_da2ad9e064cd.slice/crio-e4fecd0a6bd1066d2e6e7b3e606213740b015bfb92fab9bf6f2ae643ffe0ef21 WatchSource:0}: Error finding container e4fecd0a6bd1066d2e6e7b3e606213740b015bfb92fab9bf6f2ae643ffe0ef21: Status 404 returned error can't find the container with id e4fecd0a6bd1066d2e6e7b3e606213740b015bfb92fab9bf6f2ae643ffe0ef21 Sep 16 18:44:25.833749 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:25.833692 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f973677_db95_4065_851a_c26414a4c960.slice/crio-634928ea8c5fd30ae31166df1e8effbd12b706686482094ed159ab6c700995c9 WatchSource:0}: Error finding container 634928ea8c5fd30ae31166df1e8effbd12b706686482094ed159ab6c700995c9: Status 404 returned error can't find the container with id 634928ea8c5fd30ae31166df1e8effbd12b706686482094ed159ab6c700995c9 Sep 16 18:44:25.841054 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:25.840997 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80653a37_6288_47de_8d79_e913d7a23687.slice/crio-ef054b2028866fa14182b4d740c6759b38064ec09f59e5629df9a33ecd848b0a WatchSource:0}: Error finding container ef054b2028866fa14182b4d740c6759b38064ec09f59e5629df9a33ecd848b0a: Status 404 returned error can't find the container with id ef054b2028866fa14182b4d740c6759b38064ec09f59e5629df9a33ecd848b0a Sep 16 18:44:25.842958 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:25.842940 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b647a36_5a97_423e_af52_e5cf39fa9a1d.slice/crio-35ae7f3232ee70bbdcd51894c212597278687ecbcdf49ace100dd62fbb416e91 WatchSource:0}: Error finding container 35ae7f3232ee70bbdcd51894c212597278687ecbcdf49ace100dd62fbb416e91: Status 404 returned error can't find the container with id 35ae7f3232ee70bbdcd51894c212597278687ecbcdf49ace100dd62fbb416e91 Sep 16 18:44:25.844132 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:25.844104 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1add103_27f0_42c2_b4c2_0b96269754fe.slice/crio-8f1713ba8377e81d7386f7a7a5419177abc1e3d0da18612af498f25c8da38925 WatchSource:0}: Error finding container 8f1713ba8377e81d7386f7a7a5419177abc1e3d0da18612af498f25c8da38925: Status 404 returned error can't find the container with id 8f1713ba8377e81d7386f7a7a5419177abc1e3d0da18612af498f25c8da38925 Sep 16 18:44:25.863454 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.863425 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:24 +0000 UTC" deadline="2028-05-07 04:48:30.29867344 +0000 UTC" Sep 16 18:44:25.863454 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.863453 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14362h4m4.435224158s" Sep 16 18:44:25.877961 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.877935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:25.878072 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.877947 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:25.878072 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.878053 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:25.878182 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:25.878142 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:25.887182 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.887129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-htfgb" event={"ID":"4b647a36-5a97-423e-af52-e5cf39fa9a1d","Type":"ContainerStarted","Data":"35ae7f3232ee70bbdcd51894c212597278687ecbcdf49ace100dd62fbb416e91"} Sep 16 18:44:25.888374 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.888346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h9qbx" event={"ID":"d1add103-27f0-42c2-b4c2-0b96269754fe","Type":"ContainerStarted","Data":"8f1713ba8377e81d7386f7a7a5419177abc1e3d0da18612af498f25c8da38925"} Sep 16 18:44:25.889431 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.889410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rcnm2" event={"ID":"23954e0d-2798-4c79-8b43-244302141d00","Type":"ContainerStarted","Data":"502ec5af7b49cf38f83855d22392d79470fbe8953cd39d6d7f64136aa93fee56"} Sep 16 18:44:25.890449 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.890426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4p8g" event={"ID":"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1","Type":"ContainerStarted","Data":"e22f3643df6e96642394f31ab817033890063d1c532dd5b8825d4fb7e659863e"} Sep 16 18:44:25.891662 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.891640 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p72gp" event={"ID":"321dd925-57a6-48a7-b8b4-da2ad9e064cd","Type":"ContainerStarted","Data":"e4fecd0a6bd1066d2e6e7b3e606213740b015bfb92fab9bf6f2ae643ffe0ef21"} Sep 16 18:44:25.892897 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.892874 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"cc4650fd74b7ed3d51e0cd63bd4981fff13a6a5fceabd61bd09420c3138848c2"} Sep 16 18:44:25.894258 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.894231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwbzs" event={"ID":"80653a37-6288-47de-8d79-e913d7a23687","Type":"ContainerStarted","Data":"ef054b2028866fa14182b4d740c6759b38064ec09f59e5629df9a33ecd848b0a"} Sep 16 18:44:25.895998 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.895973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" event={"ID":"59781a33-3f1e-46cc-864e-a57b0e76fc67","Type":"ContainerStarted","Data":"720856407e8ddc2cc3a4f219107ca9237fa43139f99133ea1e0b4e2f6a133e40"} Sep 16 18:44:25.897210 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:25.897174 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerStarted","Data":"634928ea8c5fd30ae31166df1e8effbd12b706686482094ed159ab6c700995c9"} Sep 16 18:44:26.389493 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.389450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:26.389669 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.389538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:26.389745 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.389704 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:26.389811 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.389770 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret podName:f2e9015b-775b-4a7d-8542-9a1222e7f67b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:28.389750831 +0000 UTC m=+6.035845621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret") pod "global-pull-secret-syncer-sm66h" (UID: "f2e9015b-775b-4a7d-8542-9a1222e7f67b") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:26.390208 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.390173 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:26.390279 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.390224 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:28.39020816 +0000 UTC m=+6.036302945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:26.491628 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.490910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:26.491628 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.491144 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:26.491628 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.491170 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:26.491628 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.491185 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dtdlx for pod openshift-network-diagnostics/network-check-target-w8b6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:26.491628 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.491249 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx podName:6b6ef13b-8221-4c89-981c-d17ed779bbca nodeName:}" failed. No retries permitted until 2026-09-16 18:44:28.491229981 +0000 UTC m=+6.137324779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dtdlx" (UniqueName: "kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx") pod "network-check-target-w8b6s" (UID: "6b6ef13b-8221-4c89-981c-d17ed779bbca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:26.864742 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.864628 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:24 +0000 UTC" deadline="2028-03-11 13:05:29.292051476 +0000 UTC" Sep 16 18:44:26.864742 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.864703 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13002h21m2.427352855s" Sep 16 18:44:26.880565 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.880376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:26.880565 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:26.880500 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:26.912533 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.912499 2576 generic.go:356] "Generic (PLEG): container finished" podID="8291ccbc2b9783dfbef0924ad26b3f7f" containerID="4bdda0550fb19bed13943c9f5df5a71f213d6322b69869d7c84ceb2d587c7a81" exitCode=0 Sep 16 18:44:26.913435 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.913411 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" event={"ID":"8291ccbc2b9783dfbef0924ad26b3f7f","Type":"ContainerDied","Data":"4bdda0550fb19bed13943c9f5df5a71f213d6322b69869d7c84ceb2d587c7a81"} Sep 16 18:44:26.920749 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:26.920717 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" event={"ID":"250437f78067fb73286aae1b34e08bfc","Type":"ContainerStarted","Data":"f875ef81f28ed45ab396498f8970a6082db6f1c41d4e2e6a73d958493a20cbb9"} Sep 16 18:44:27.877969 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:27.877938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:27.878420 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:27.878059 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:27.878523 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:27.878414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:27.878586 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:27.878553 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:27.932030 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:27.931956 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" event={"ID":"8291ccbc2b9783dfbef0924ad26b3f7f","Type":"ContainerStarted","Data":"85d8c4bbc39598680d20b4a216f68fc9bf25bd298b8453b13298e6f462aae6bf"} Sep 16 18:44:27.946720 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:27.945545 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-79.ec2.internal" podStartSLOduration=3.94552417 podStartE2EDuration="3.94552417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:26.943238651 +0000 UTC m=+4.589333455" watchObservedRunningTime="2026-09-16 18:44:27.94552417 +0000 UTC m=+5.591618977" Sep 16 18:44:28.407079 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:28.406094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:28.407079 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:28.406460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:28.407079 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.406323 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:28.407079 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.406668 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret podName:f2e9015b-775b-4a7d-8542-9a1222e7f67b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.406649637 +0000 UTC m=+10.052744471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret") pod "global-pull-secret-syncer-sm66h" (UID: "f2e9015b-775b-4a7d-8542-9a1222e7f67b") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:28.407079 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.406599 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:28.407079 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.407045 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.407028272 +0000 UTC m=+10.053123072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:28.507060 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:28.507022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:28.507242 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.507224 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:28.507327 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.507251 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:28.507327 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.507265 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dtdlx for pod openshift-network-diagnostics/network-check-target-w8b6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:28.507327 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.507324 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx podName:6b6ef13b-8221-4c89-981c-d17ed779bbca nodeName:}" failed. No retries permitted until 2026-09-16 18:44:32.507306581 +0000 UTC m=+10.153401390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dtdlx" (UniqueName: "kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx") pod "network-check-target-w8b6s" (UID: "6b6ef13b-8221-4c89-981c-d17ed779bbca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:28.880427 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:28.880353 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:28.880879 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:28.880477 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:29.878224 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:29.877609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:29.878224 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:29.877732 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:29.878224 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:29.878096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:29.878224 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:29.878172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:30.878003 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:30.877967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:30.878391 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:30.878111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:31.878221 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:31.878189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:31.878683 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:31.878237 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:31.878683 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:31.878302 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:31.878683 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:31.878437 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:32.441174 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:32.441137 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:32.441323 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:32.441202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:32.441375 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.441341 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:32.441409 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.441404 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret podName:f2e9015b-775b-4a7d-8542-9a1222e7f67b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:40.441383771 +0000 UTC m=+18.087478556 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret") pod "global-pull-secret-syncer-sm66h" (UID: "f2e9015b-775b-4a7d-8542-9a1222e7f67b") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:32.441820 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.441802 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:32.441890 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.441854 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:40.441839442 +0000 UTC m=+18.087934230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:32.542383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:32.542297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:32.542570 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.542469 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:32.542570 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.542505 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:32.542570 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.542515 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dtdlx for pod openshift-network-diagnostics/network-check-target-w8b6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:32.542570 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.542567 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx podName:6b6ef13b-8221-4c89-981c-d17ed779bbca nodeName:}" failed. No retries permitted until 2026-09-16 18:44:40.542552391 +0000 UTC m=+18.188647188 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dtdlx" (UniqueName: "kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx") pod "network-check-target-w8b6s" (UID: "6b6ef13b-8221-4c89-981c-d17ed779bbca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:32.878980 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:32.878896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:32.879406 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:32.879016 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:33.877716 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:33.877678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:33.877898 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:33.877820 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:33.878256 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:33.878238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:33.878368 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:33.878341 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:34.878007 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:34.877974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:34.878407 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:34.878108 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:35.877441 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:35.877407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:35.877623 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:35.877411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:35.877623 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:35.877568 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:35.877755 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:35.877727 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:36.878238 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:36.878162 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:36.878685 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:36.878283 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:37.878035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:37.878004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:37.878215 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:37.878128 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:37.878439 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:37.878001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:37.878439 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:37.878394 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:38.877800 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:38.877766 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:38.877976 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:38.877871 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:39.878284 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:39.878248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:39.878729 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:39.878349 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:39.878729 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:39.878403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:39.878729 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:39.878517 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:40.504265 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:40.504186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:40.504265 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:40.504241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:40.504465 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.504351 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:40.504465 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.504376 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:40.504465 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.504423 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:56.504404015 +0000 UTC m=+34.150498802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:40.504465 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.504440 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret podName:f2e9015b-775b-4a7d-8542-9a1222e7f67b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:56.50443324 +0000 UTC m=+34.150528028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret") pod "global-pull-secret-syncer-sm66h" (UID: "f2e9015b-775b-4a7d-8542-9a1222e7f67b") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:40.604855 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:40.604812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:40.605049 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.604963 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:40.605049 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.604991 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:40.605049 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.605004 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dtdlx for pod openshift-network-diagnostics/network-check-target-w8b6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:40.605207 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.605063 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx podName:6b6ef13b-8221-4c89-981c-d17ed779bbca nodeName:}" failed. No retries permitted until 2026-09-16 18:44:56.605048816 +0000 UTC m=+34.251143600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dtdlx" (UniqueName: "kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx") pod "network-check-target-w8b6s" (UID: "6b6ef13b-8221-4c89-981c-d17ed779bbca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:40.878460 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:40.877911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:40.878460 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:40.878045 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:41.877973 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:41.877939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:41.878146 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:41.877939 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:41.878146 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:41.878061 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:41.878265 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:41.878144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:42.878801 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:42.878772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:42.879193 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:42.878858 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:43.878265 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:43.878236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:43.878422 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:43.878326 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:43.878422 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:43.878378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:43.878541 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:43.878465 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:44.878631 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.878422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:44.879415 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:44.878728 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:44.965363 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.965291 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-htfgb" event={"ID":"4b647a36-5a97-423e-af52-e5cf39fa9a1d","Type":"ContainerStarted","Data":"9945cfaba13f79f1ad43a659bb25c9b4648f71c1cecce6db4dfc799d66267790"} Sep 16 18:44:44.966937 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.966916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rcnm2" event={"ID":"23954e0d-2798-4c79-8b43-244302141d00","Type":"ContainerStarted","Data":"b3519346534f1371ca61e822f033742d4e8b8d23ca59c9122f4b70eb08eacb43"} Sep 16 18:44:44.968320 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.968302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4p8g" event={"ID":"2e46d10a-d3c7-4b28-bab1-a16ffbe596b1","Type":"ContainerStarted","Data":"0b21778f9ff2f659613b9e25d85e60037d1c458894263ba1fbe7830886947640"} Sep 16 18:44:44.969849 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.969828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p72gp" event={"ID":"321dd925-57a6-48a7-b8b4-da2ad9e064cd","Type":"ContainerStarted","Data":"b4bc507d9d368d528f1b860ad075c2d5d23976e839fe348bea47ed070225535a"} Sep 16 18:44:44.972417 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972402 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:44:44.972727 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972712 2576 generic.go:356] "Generic (PLEG): container finished" podID="c6d71861-fc83-4426-9f3e-15ca25b91f00" containerID="dd55e0283f50b385ef241fd29e4556a6b7ea976871f1639e1e1842ed70af4b8f" exitCode=1 Sep 16 18:44:44.972785 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972770 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"f2009add8a4f080b77326e4ca7e68c877df5132debf8648a55af9fa263f39fab"} Sep 16 18:44:44.972834 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972823 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"11c0e1ecf63b77835253003bec0c11c27fe17f0b677e3deab7e80f345f65b256"} Sep 16 18:44:44.972876 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"c509dc8215bc19a16c6d74dfff1dc63d5e21f6ded22d1244b32605cbcb3f6822"} Sep 16 18:44:44.972911 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"e3893e178dfbe1ee0337177ac44763ed5c71b38b7ca14e543b3d8ca98ca2dd2b"} Sep 16 18:44:44.972944 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerDied","Data":"dd55e0283f50b385ef241fd29e4556a6b7ea976871f1639e1e1842ed70af4b8f"} Sep 16 18:44:44.972980 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.972974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"ddf93faed987478f32e165a5a6e9b109425b1eedd81436f7a1332efe19bd4075"} Sep 16 18:44:44.974136 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.974117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dwbzs" event={"ID":"80653a37-6288-47de-8d79-e913d7a23687","Type":"ContainerStarted","Data":"60c2dd0556c326450efa26787cb2ea959637d6902d527ec4bfef2376d78c0676"} Sep 16 18:44:44.975410 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.975390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" event={"ID":"59781a33-3f1e-46cc-864e-a57b0e76fc67","Type":"ContainerStarted","Data":"21543eb6636e7d7ddefa78ab834cb5693824b6a6a5586b0e7dc1d639aa4f1b0f"} Sep 16 18:44:44.976715 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.976693 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f973677-db95-4065-851a-c26414a4c960" containerID="9fd0136714ab66342f7f68f4b156720da854d5e4ec3659060b5aad659d62e342" exitCode=0 Sep 16 18:44:44.976786 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.976723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerDied","Data":"9fd0136714ab66342f7f68f4b156720da854d5e4ec3659060b5aad659d62e342"} Sep 16 18:44:44.990338 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.990297 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-htfgb" podStartSLOduration=3.924261602 podStartE2EDuration="21.99028132s" totalImagesPullingTime="18.066019718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:44.990055605 +0000 UTC m=+22.636150411" watchObservedRunningTime="2026-09-16 18:44:44.99028132 +0000 UTC m=+22.636376137" Sep 16 18:44:44.990446 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:44.990405 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-79.ec2.internal" podStartSLOduration=20.990398869 podStartE2EDuration="20.990398869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:27.946693878 +0000 UTC m=+5.592788686" watchObservedRunningTime="2026-09-16 18:44:44.990398869 +0000 UTC m=+22.636493675" Sep 16 18:44:45.010564 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.010517 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p72gp" podStartSLOduration=3.865909953 podStartE2EDuration="22.010498489s" totalImagesPullingTime="18.144588536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:45.01040902 +0000 UTC m=+22.656503836" watchObservedRunningTime="2026-09-16 18:44:45.010498489 +0000 UTC m=+22.656593290" Sep 16 18:44:45.073185 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.073132 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q4p8g" podStartSLOduration=3.9301241449999997 podStartE2EDuration="22.073111892s" totalImagesPullingTime="18.142987747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:45.072738393 +0000 UTC m=+22.718833213" watchObservedRunningTime="2026-09-16 18:44:45.073111892 +0000 UTC m=+22.719206699" Sep 16 18:44:45.109774 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.109718 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dwbzs" podStartSLOduration=3.350081353 podStartE2EDuration="22.109694764s" totalImagesPullingTime="18.759613411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:45.109006577 +0000 UTC m=+22.755101384" watchObservedRunningTime="2026-09-16 18:44:45.109694764 +0000 UTC m=+22.755789573" Sep 16 18:44:45.131831 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.131775 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rcnm2" podStartSLOduration=4.003762827 podStartE2EDuration="22.131754856s" totalImagesPullingTime="18.127992029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:45.13080496 +0000 UTC m=+22.776899761" watchObservedRunningTime="2026-09-16 18:44:45.131754856 +0000 UTC m=+22.777849663" Sep 16 18:44:45.321909 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.321879 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:45.322571 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.322552 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:45.723135 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.722994 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 18:44:45.836419 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.836237 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:44:45.72312934Z","UUID":"8d353e6c-7e36-4589-af2a-fa6cc2faaf3a","Handler":null,"Name":"","Endpoint":""} Sep 16 18:44:45.838178 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.838146 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 18:44:45.838178 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.838179 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 18:44:45.878260 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.878226 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:45.878401 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:45.878323 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:45.878473 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.878405 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:45.878582 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:45.878545 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:45.981154 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.981120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-h9qbx" event={"ID":"d1add103-27f0-42c2-b4c2-0b96269754fe","Type":"ContainerStarted","Data":"24f24376c746d24a4843b6ba6fcc41ab44f8b5dbf79d00adaeea875e28155dd4"} Sep 16 18:44:45.983556 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:45.983521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" event={"ID":"59781a33-3f1e-46cc-864e-a57b0e76fc67","Type":"ContainerStarted","Data":"6a567896ad9c4ec08ca8c39642c44ca9ce48d6c9cdee66a2ae7006de7b9931ce"} Sep 16 18:44:46.877965 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:46.877338 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:46.877965 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:46.877640 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:46.988757 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:46.988605 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:44:46.989131 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:46.988997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"206b776e795fcb77d096da81ed0bac88f6ead5e7709225f9b92ab0bf2ac68269"} Sep 16 18:44:46.989131 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:46.989122 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 16 18:44:47.877576 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:47.877545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:47.877768 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:47.877556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:47.877768 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:47.877699 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:47.877768 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:47.877746 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:47.993750 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:47.993718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" event={"ID":"59781a33-3f1e-46cc-864e-a57b0e76fc67","Type":"ContainerStarted","Data":"4db99acc179a2d5c99d084b075d46d36d2d9a5770db773a5e27b6df02bb3ecba"} Sep 16 18:44:48.015338 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:48.015287 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-h9qbx" podStartSLOduration=6.951365205 podStartE2EDuration="25.015266927s" totalImagesPullingTime="18.063901722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:45.998991163 +0000 UTC m=+23.645085971" watchObservedRunningTime="2026-09-16 18:44:48.015266927 +0000 UTC m=+25.661361734" Sep 16 18:44:48.015546 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:48.015400 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jhkv4" podStartSLOduration=4.406884551 podStartE2EDuration="25.015393273s" totalImagesPullingTime="20.608508722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:48.014634773 +0000 UTC m=+25.660729590" watchObservedRunningTime="2026-09-16 18:44:48.015393273 +0000 UTC m=+25.661488080" Sep 16 18:44:48.877599 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:48.877570 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:48.877789 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:48.877693 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:49.878359 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:49.878164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:49.878723 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:49.878164 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:49.878723 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:49.878384 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:49.878723 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:49.878470 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:50.001537 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.001512 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:44:50.001897 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.001867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"e5619809f52c278dbdade6777c3be84fb2d190b8dad800faf23c6bc5b672dc6c"} Sep 16 18:44:50.002161 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.002126 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:50.002348 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.002331 2576 scope.go:117] "RemoveContainer" containerID="dd55e0283f50b385ef241fd29e4556a6b7ea976871f1639e1e1842ed70af4b8f" Sep 16 18:44:50.003864 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.003842 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f973677-db95-4065-851a-c26414a4c960" containerID="bd84faebb115d39a727bcac43233f0f9b1dba5d3f3f4a985726fff8fa9bdd5e6" exitCode=0 Sep 16 18:44:50.003948 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.003879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerDied","Data":"bd84faebb115d39a727bcac43233f0f9b1dba5d3f3f4a985726fff8fa9bdd5e6"} Sep 16 18:44:50.019307 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.019287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:50.878324 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:50.878301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:50.878569 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:50.878408 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:51.009062 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.009035 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:44:51.009380 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.009357 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" event={"ID":"c6d71861-fc83-4426-9f3e-15ca25b91f00","Type":"ContainerStarted","Data":"5ce40549e4ec0b67b95639fa490fb7fea96693cd89e17c3a6bcaee1c3731238d"} Sep 16 18:44:51.009693 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.009667 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:51.009795 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.009730 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:51.025518 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.025499 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:44:51.043306 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.043269 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" podStartSLOduration=9.602042484 podStartE2EDuration="28.043255942s" totalImagesPullingTime="18.441213458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:51.043236639 +0000 UTC m=+28.689331447" watchObservedRunningTime="2026-09-16 18:44:51.043255942 +0000 UTC m=+28.689350749" Sep 16 18:44:51.045762 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.045741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm66h"] Sep 16 18:44:51.045866 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.045853 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:51.045983 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:51.045965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:51.049124 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.049103 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b47hq"] Sep 16 18:44:51.049206 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.049165 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w8b6s"] Sep 16 18:44:51.049266 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.049211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:51.049315 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:51.049295 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:51.049363 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:51.049333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:51.049476 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:51.049455 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:52.013301 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:52.013263 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f973677-db95-4065-851a-c26414a4c960" containerID="9a67535f9e69883f8ceac1b46ea24a398aed07f3bd5624a670e8da5f3c27662c" exitCode=0 Sep 16 18:44:52.013773 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:52.013346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerDied","Data":"9a67535f9e69883f8ceac1b46ea24a398aed07f3bd5624a670e8da5f3c27662c"} Sep 16 18:44:52.878269 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:52.878245 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:52.878388 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:52.878343 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:52.878388 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:52.878372 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:52.878535 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:52.878473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:52.878535 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:52.878524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:52.878663 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:52.878637 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:53.018371 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:53.018340 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f973677-db95-4065-851a-c26414a4c960" containerID="17f0366b4d173ff59a96bca2c431480b38d623c43359f076308ccaa48c0ee271" exitCode=0 Sep 16 18:44:53.018816 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:53.018429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerDied","Data":"17f0366b4d173ff59a96bca2c431480b38d623c43359f076308ccaa48c0ee271"} Sep 16 18:44:54.430660 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:54.430419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:54.431082 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:54.430772 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 16 18:44:54.431082 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:54.431072 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-htfgb" Sep 16 18:44:54.881207 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:54.881176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:54.881359 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:54.881216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:54.881359 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:54.881177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:54.881359 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:54.881299 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:54.881578 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:54.881407 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:54.881578 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:54.881455 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:56.517910 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:56.517872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:56.518323 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:56.517926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:56.518323 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.518013 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:56.518323 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.518029 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:56.518323 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.518076 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret podName:f2e9015b-775b-4a7d-8542-9a1222e7f67b nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.518057769 +0000 UTC m=+66.164152558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret") pod "global-pull-secret-syncer-sm66h" (UID: "f2e9015b-775b-4a7d-8542-9a1222e7f67b") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:44:56.518323 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.518098 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.518087706 +0000 UTC m=+66.164182496 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:44:56.618907 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:56.618872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:56.619076 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.619002 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:44:56.619076 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.619022 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:44:56.619076 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.619034 2576 projected.go:196] Error preparing data for projected volume kube-api-access-dtdlx for pod openshift-network-diagnostics/network-check-target-w8b6s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:56.619213 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.619094 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx podName:6b6ef13b-8221-4c89-981c-d17ed779bbca nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.619079543 +0000 UTC m=+66.265174327 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dtdlx" (UniqueName: "kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx") pod "network-check-target-w8b6s" (UID: "6b6ef13b-8221-4c89-981c-d17ed779bbca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:44:56.881646 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:56.881619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:56.881807 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:56.881732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:56.881807 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.881738 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sm66h" podUID="f2e9015b-775b-4a7d-8542-9a1222e7f67b" Sep 16 18:44:56.881908 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.881859 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w8b6s" podUID="6b6ef13b-8221-4c89-981c-d17ed779bbca" Sep 16 18:44:56.881908 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:56.881889 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:56.882004 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:56.881981 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b47hq" podUID="e3fd2611-8884-4a84-861e-1342cee39e99" Sep 16 18:44:57.036063 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.036033 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerStarted","Data":"d9a738f12cc3240b0103d672125f7d657fa24a1027ed092fca7710155f5dc5b1"} Sep 16 18:44:57.074994 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.074973 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-79.ec2.internal" event="NodeReady" Sep 16 18:44:57.075100 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.075090 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 18:44:57.137379 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.137347 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f9cc85bb7-qdbbt"] Sep 16 18:44:57.139234 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.139219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.142260 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.142096 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 18:44:57.142260 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.142131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 18:44:57.142466 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.142423 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 18:44:57.142564 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.142530 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7kp4l\"" Sep 16 18:44:57.146446 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.146429 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd"] Sep 16 18:44:57.150673 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.150329 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf"] Sep 16 18:44:57.150673 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.150417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.152667 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.152641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 18:44:57.152838 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.152823 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 18:44:57.153298 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.153279 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t"] Sep 16 18:44:57.153432 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.153416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.153536 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.153521 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 18:44:57.153720 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.153700 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 18:44:57.153720 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.153823 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 18:44:57.155529 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.155513 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.156849 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.156829 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 16 18:44:57.156988 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.156966 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 16 18:44:57.157370 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.157325 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 16 18:44:57.157593 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.157576 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 16 18:44:57.157901 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.157887 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 18:44:57.157941 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.157927 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-72nrx\"" Sep 16 18:44:57.160190 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.160169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd"] Sep 16 18:44:57.162773 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.162755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t"] Sep 16 18:44:57.172467 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.172448 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8c758"] Sep 16 18:44:57.174766 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.174748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.177849 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.177829 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 18:44:57.178033 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.177998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 18:44:57.178190 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.178156 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6fbqp"] Sep 16 18:44:57.179448 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.179432 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g84ch\"" Sep 16 18:44:57.179908 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.179892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.185710 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.185695 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 18:44:57.186990 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.186974 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 18:44:57.187185 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.187173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9lmtn\"" Sep 16 18:44:57.188042 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.188027 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 18:44:57.200968 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.200950 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8c758"] Sep 16 18:44:57.206713 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.206695 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6fbqp"] Sep 16 18:44:57.222032 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r799c\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-kube-api-access-r799c\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222198 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmj8\" (UniqueName: \"kubernetes.io/projected/3ac554d4-1b23-4222-82e9-129d22e14e53-kube-api-access-wmmj8\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.222198 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-registry-certificates\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222198 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdszj\" (UniqueName: \"kubernetes.io/projected/ceb10211-4c66-4d9f-b69b-00de326fd277-kube-api-access-fdszj\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.222198 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222128 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32cfcfbb-1474-47b0-8f03-4e5ade92a415-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b89554f66-zm44t\" (UID: \"32cfcfbb-1474-47b0-8f03-4e5ade92a415\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.222198 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222152 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceb10211-4c66-4d9f-b69b-00de326fd277-tmp\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.222198 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222169 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-ca\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-installation-pull-secrets\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-image-registry-private-configuration\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5fc\" (UniqueName: \"kubernetes.io/projected/32cfcfbb-1474-47b0-8f03-4e5ade92a415-kube-api-access-4h5fc\") pod \"managed-serviceaccount-addon-agent-5b89554f66-zm44t\" (UID: \"32cfcfbb-1474-47b0-8f03-4e5ade92a415\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-trusted-ca\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222416 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac554d4-1b23-4222-82e9-129d22e14e53-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.222696 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ceb10211-4c66-4d9f-b69b-00de326fd277-klusterlet-config\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.222696 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-bound-sa-token\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222696 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3fc0590-a623-4573-98f6-c668b336b05c-ca-trust-extracted\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.222696 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.222517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-hub\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.224729 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.224704 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f9cc85bb7-qdbbt"] Sep 16 18:44:57.278756 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.278724 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf"] Sep 16 18:44:57.323306 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-registry-certificates\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.323306 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdszj\" (UniqueName: \"kubernetes.io/projected/ceb10211-4c66-4d9f-b69b-00de326fd277-kube-api-access-fdszj\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32cfcfbb-1474-47b0-8f03-4e5ade92a415-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b89554f66-zm44t\" (UID: \"32cfcfbb-1474-47b0-8f03-4e5ade92a415\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceb10211-4c66-4d9f-b69b-00de326fd277-tmp\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-ca\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323458 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-installation-pull-secrets\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-image-registry-private-configuration\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5fc\" (UniqueName: \"kubernetes.io/projected/32cfcfbb-1474-47b0-8f03-4e5ade92a415-kube-api-access-4h5fc\") pod \"managed-serviceaccount-addon-agent-5b89554f66-zm44t\" (UID: \"32cfcfbb-1474-47b0-8f03-4e5ade92a415\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.323555 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smqjb\" (UniqueName: \"kubernetes.io/projected/77417086-a1e1-4b6a-b8fe-b168e3c07c59-kube-api-access-smqjb\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-trusted-ca\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac554d4-1b23-4222-82e9-129d22e14e53-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77417086-a1e1-4b6a-b8fe-b168e3c07c59-config-volume\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ceb10211-4c66-4d9f-b69b-00de326fd277-klusterlet-config\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-bound-sa-token\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.323814 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.323834 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.323897 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:57.823878142 +0000 UTC m=+35.469972946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77417086-a1e1-4b6a-b8fe-b168e3c07c59-tmp-dir\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324155 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3fc0590-a623-4573-98f6-c668b336b05c-ca-trust-extracted\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-hub\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r799c\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-kube-api-access-r799c\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324250 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmj8\" (UniqueName: \"kubernetes.io/projected/3ac554d4-1b23-4222-82e9-129d22e14e53-kube-api-access-wmmj8\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.325223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324279 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e38d3-3525-407d-9e6c-06aedad02c7a-cert\") pod \"ingress-canary-6fbqp\" (UID: \"735e38d3-3525-407d-9e6c-06aedad02c7a\") " pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.326119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht8ms\" (UniqueName: \"kubernetes.io/projected/735e38d3-3525-407d-9e6c-06aedad02c7a-kube-api-access-ht8ms\") pod \"ingress-canary-6fbqp\" (UID: \"735e38d3-3525-407d-9e6c-06aedad02c7a\") " pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.326119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324582 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-registry-certificates\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.326119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.324689 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3fc0590-a623-4573-98f6-c668b336b05c-ca-trust-extracted\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.326119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.325402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-trusted-ca\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.326119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.323797 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ceb10211-4c66-4d9f-b69b-00de326fd277-tmp\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.326545 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.326473 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ac554d4-1b23-4222-82e9-129d22e14e53-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.332527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.329334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-image-registry-private-configuration\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.332527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.329419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-installation-pull-secrets\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.332527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.329572 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-hub\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.332527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.329581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-ca\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.332527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.329589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/32cfcfbb-1474-47b0-8f03-4e5ade92a415-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b89554f66-zm44t\" (UID: \"32cfcfbb-1474-47b0-8f03-4e5ade92a415\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.332527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.329623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.332527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.329626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ac554d4-1b23-4222-82e9-129d22e14e53-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.333650 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.333629 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ceb10211-4c66-4d9f-b69b-00de326fd277-klusterlet-config\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.334733 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.334714 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5fc\" (UniqueName: \"kubernetes.io/projected/32cfcfbb-1474-47b0-8f03-4e5ade92a415-kube-api-access-4h5fc\") pod \"managed-serviceaccount-addon-agent-5b89554f66-zm44t\" (UID: \"32cfcfbb-1474-47b0-8f03-4e5ade92a415\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.335279 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.335255 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdszj\" (UniqueName: \"kubernetes.io/projected/ceb10211-4c66-4d9f-b69b-00de326fd277-kube-api-access-fdszj\") pod \"klusterlet-addon-workmgr-549ff96c69-9pnqd\" (UID: \"ceb10211-4c66-4d9f-b69b-00de326fd277\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.336422 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.336405 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r799c\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-kube-api-access-r799c\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.337617 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.337595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-bound-sa-token\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.338804 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.338787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmj8\" (UniqueName: \"kubernetes.io/projected/3ac554d4-1b23-4222-82e9-129d22e14e53-kube-api-access-wmmj8\") pod \"cluster-proxy-proxy-agent-9b4695f46-4nszf\" (UID: \"3ac554d4-1b23-4222-82e9-129d22e14e53\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.407387 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.407310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:44:57.425373 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.425347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e38d3-3525-407d-9e6c-06aedad02c7a-cert\") pod \"ingress-canary-6fbqp\" (UID: \"735e38d3-3525-407d-9e6c-06aedad02c7a\") " pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.425440 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.425378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht8ms\" (UniqueName: \"kubernetes.io/projected/735e38d3-3525-407d-9e6c-06aedad02c7a-kube-api-access-ht8ms\") pod \"ingress-canary-6fbqp\" (UID: \"735e38d3-3525-407d-9e6c-06aedad02c7a\") " pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.425513 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.425453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smqjb\" (UniqueName: \"kubernetes.io/projected/77417086-a1e1-4b6a-b8fe-b168e3c07c59-kube-api-access-smqjb\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.425559 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.425515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77417086-a1e1-4b6a-b8fe-b168e3c07c59-config-volume\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.425559 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.425542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.425644 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.425567 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77417086-a1e1-4b6a-b8fe-b168e3c07c59-tmp-dir\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.425712 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.425694 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:44:57.425763 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.425752 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:57.925737891 +0000 UTC m=+35.571832679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:44:57.425880 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.425857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77417086-a1e1-4b6a-b8fe-b168e3c07c59-tmp-dir\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.426122 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.426101 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77417086-a1e1-4b6a-b8fe-b168e3c07c59-config-volume\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.427985 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.427971 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/735e38d3-3525-407d-9e6c-06aedad02c7a-cert\") pod \"ingress-canary-6fbqp\" (UID: \"735e38d3-3525-407d-9e6c-06aedad02c7a\") " pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.437724 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.437698 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht8ms\" (UniqueName: \"kubernetes.io/projected/735e38d3-3525-407d-9e6c-06aedad02c7a-kube-api-access-ht8ms\") pod \"ingress-canary-6fbqp\" (UID: \"735e38d3-3525-407d-9e6c-06aedad02c7a\") " pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.438255 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.438236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smqjb\" (UniqueName: \"kubernetes.io/projected/77417086-a1e1-4b6a-b8fe-b168e3c07c59-kube-api-access-smqjb\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.460855 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.460837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:44:57.474547 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.474530 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:44:57.491853 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.491630 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" Sep 16 18:44:57.501761 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.501734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6fbqp" Sep 16 18:44:57.516413 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.516322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:44:57.748685 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.748665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd"] Sep 16 18:44:57.751758 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:57.751734 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb10211_4c66_4d9f_b69b_00de326fd277.slice/crio-dc1992754a5df89e9dc1dca9a7191dc9e9da7213d04f533daa217f2d058055fd WatchSource:0}: Error finding container dc1992754a5df89e9dc1dca9a7191dc9e9da7213d04f533daa217f2d058055fd: Status 404 returned error can't find the container with id dc1992754a5df89e9dc1dca9a7191dc9e9da7213d04f533daa217f2d058055fd Sep 16 18:44:57.760599 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.760572 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf"] Sep 16 18:44:57.763281 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:57.763257 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ac554d4_1b23_4222_82e9_129d22e14e53.slice/crio-3b4de4ef4582cdae3c96e2677dc176b7f4891ce4bdde7057a93c1d08ceed61e3 WatchSource:0}: Error finding container 3b4de4ef4582cdae3c96e2677dc176b7f4891ce4bdde7057a93c1d08ceed61e3: Status 404 returned error can't find the container with id 3b4de4ef4582cdae3c96e2677dc176b7f4891ce4bdde7057a93c1d08ceed61e3 Sep 16 18:44:57.780820 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.780788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t"] Sep 16 18:44:57.784184 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:57.784162 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32cfcfbb_1474_47b0_8f03_4e5ade92a415.slice/crio-60d77a90cb58c1a03960d5f54afbb86b13eed4f83e8ba4a6197cc6e38de2b8d8 WatchSource:0}: Error finding container 60d77a90cb58c1a03960d5f54afbb86b13eed4f83e8ba4a6197cc6e38de2b8d8: Status 404 returned error can't find the container with id 60d77a90cb58c1a03960d5f54afbb86b13eed4f83e8ba4a6197cc6e38de2b8d8 Sep 16 18:44:57.800882 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.800857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6fbqp"] Sep 16 18:44:57.803314 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:44:57.803289 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735e38d3_3525_407d_9e6c_06aedad02c7a.slice/crio-c83328420e265d8836c11cb7a8ab2052af93c7455e764c1a6a8b4e068c1b6514 WatchSource:0}: Error finding container c83328420e265d8836c11cb7a8ab2052af93c7455e764c1a6a8b4e068c1b6514: Status 404 returned error can't find the container with id c83328420e265d8836c11cb7a8ab2052af93c7455e764c1a6a8b4e068c1b6514 Sep 16 18:44:57.829223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.829199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:57.829358 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.829341 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:44:57.829402 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.829360 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:44:57.829440 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.829407 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:58.829393914 +0000 UTC m=+36.475488698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:44:57.930409 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:57.930378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:57.930556 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.930515 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:44:57.930597 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:57.930563 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:58.930550679 +0000 UTC m=+36.576645463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:44:58.039207 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.039124 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" event={"ID":"32cfcfbb-1474-47b0-8f03-4e5ade92a415","Type":"ContainerStarted","Data":"60d77a90cb58c1a03960d5f54afbb86b13eed4f83e8ba4a6197cc6e38de2b8d8"} Sep 16 18:44:58.040316 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.040287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" event={"ID":"3ac554d4-1b23-4222-82e9-129d22e14e53","Type":"ContainerStarted","Data":"3b4de4ef4582cdae3c96e2677dc176b7f4891ce4bdde7057a93c1d08ceed61e3"} Sep 16 18:44:58.042842 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.042822 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f973677-db95-4065-851a-c26414a4c960" containerID="d9a738f12cc3240b0103d672125f7d657fa24a1027ed092fca7710155f5dc5b1" exitCode=0 Sep 16 18:44:58.042942 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.042905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerDied","Data":"d9a738f12cc3240b0103d672125f7d657fa24a1027ed092fca7710155f5dc5b1"} Sep 16 18:44:58.044097 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.043931 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6fbqp" event={"ID":"735e38d3-3525-407d-9e6c-06aedad02c7a","Type":"ContainerStarted","Data":"c83328420e265d8836c11cb7a8ab2052af93c7455e764c1a6a8b4e068c1b6514"} Sep 16 18:44:58.045115 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.045097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" event={"ID":"ceb10211-4c66-4d9f-b69b-00de326fd277","Type":"ContainerStarted","Data":"dc1992754a5df89e9dc1dca9a7191dc9e9da7213d04f533daa217f2d058055fd"} Sep 16 18:44:58.838622 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.838520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:44:58.839411 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:58.838691 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:44:58.839411 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:58.838709 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:44:58.839411 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:58.838770 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.83875241 +0000 UTC m=+38.484847197 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:44:58.882281 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.878008 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:44:58.882281 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.878463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:44:58.882281 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.878924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:44:58.882899 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.882706 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:44:58.885418 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.885094 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:44:58.885418 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.885301 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:44:58.887985 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.886104 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:44:58.887985 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.886379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pf72n\"" Sep 16 18:44:58.887985 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.886696 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dgmmm\"" Sep 16 18:44:58.939996 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:58.939961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:44:58.940253 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:58.940234 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:44:58.940338 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:44:58.940301 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.9402823 +0000 UTC m=+38.586377098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:44:59.058985 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:59.058095 2576 generic.go:356] "Generic (PLEG): container finished" podID="5f973677-db95-4065-851a-c26414a4c960" containerID="6d0a1c0c82a557cf16af1ffcc5ae2ba54ffbe3b8a2a6ab192797669f963a06a7" exitCode=0 Sep 16 18:44:59.058985 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:59.058175 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerDied","Data":"6d0a1c0c82a557cf16af1ffcc5ae2ba54ffbe3b8a2a6ab192797669f963a06a7"} Sep 16 18:44:59.251382 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:59.251325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:44:59.348275 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:44:59.348243 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:00.069216 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:00.069182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" event={"ID":"5f973677-db95-4065-851a-c26414a4c960","Type":"ContainerStarted","Data":"d61dc95e141b6b005adf4d1a59f13d0176d6b5aad70cc4d4bf04a78a01ee5286"} Sep 16 18:45:00.101934 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:00.101881 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ldjt8" podStartSLOduration=9.148142381 podStartE2EDuration="37.101857863s" totalImagesPullingTime="27.953715482s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:00.100353179 +0000 UTC m=+37.746447998" watchObservedRunningTime="2026-09-16 18:45:00.101857863 +0000 UTC m=+37.747952670" Sep 16 18:45:00.254695 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:00.254386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:00.348953 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:00.348753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:00.858235 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:00.858107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:45:00.858383 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:00.858261 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:00.858383 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:00.858279 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:45:00.858383 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:00.858337 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:04.858319112 +0000 UTC m=+42.504413909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:45:00.959660 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:00.959590 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:45:00.960087 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:00.959737 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:00.960087 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:00.959793 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:04.959775113 +0000 UTC m=+42.605869906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:45:02.870996 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:02.870959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:02.941561 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:02.941509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:04.865926 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:04.865891 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:04.891653 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:04.891617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:45:04.891803 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:04.891739 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:04.891803 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:04.891756 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:45:04.891803 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:04.891804 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:12.891790769 +0000 UTC m=+50.537885553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:45:04.967117 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:04.967083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:04.991972 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:04.991935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:45:04.992123 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:04.992087 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:04.992194 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:04.992158 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:12.992137514 +0000 UTC m=+50.638232323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:45:07.090995 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.090900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" event={"ID":"ceb10211-4c66-4d9f-b69b-00de326fd277","Type":"ContainerStarted","Data":"dc34444e605d2834215f5d505574c18f57ff53431ceecd30a7aba95c94e12f8f"} Sep 16 18:45:07.091531 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.091051 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:45:07.092586 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.092555 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" event={"ID":"32cfcfbb-1474-47b0-8f03-4e5ade92a415","Type":"ContainerStarted","Data":"ca188ea3bdaa6edbab4523a48981ae1a24ba6615b5355b05de3a80280fe66564"} Sep 16 18:45:07.092942 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.092923 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" Sep 16 18:45:07.094847 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.094819 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" event={"ID":"3ac554d4-1b23-4222-82e9-129d22e14e53","Type":"ContainerStarted","Data":"c2b3f10d316b3198edc6c637a24360e9f31eae9ca2e69b49b20811932a40b96e"} Sep 16 18:45:07.094967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.094856 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" event={"ID":"3ac554d4-1b23-4222-82e9-129d22e14e53","Type":"ContainerStarted","Data":"6bd57c556e15e05617df45c34bfbcbde056836215a488f1ce9809a599c49dcf1"} Sep 16 18:45:07.094967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.094869 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" event={"ID":"3ac554d4-1b23-4222-82e9-129d22e14e53","Type":"ContainerStarted","Data":"af4dce2e08c929df2d5ce2a8bb05ee1e12ed6ce8546966ad703eabe72367eeb0"} Sep 16 18:45:07.096195 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.096173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6fbqp" event={"ID":"735e38d3-3525-407d-9e6c-06aedad02c7a","Type":"ContainerStarted","Data":"6b132c9942962d10c6449e52d47e3c11bc8006d34720174a5a9e7b5f1b9492e7"} Sep 16 18:45:07.110531 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.110474 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-549ff96c69-9pnqd" podStartSLOduration=12.444778974 podStartE2EDuration="21.110462412s" totalImagesPullingTime="8.665683438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:07.109562478 +0000 UTC m=+44.755657284" watchObservedRunningTime="2026-09-16 18:45:07.110462412 +0000 UTC m=+44.756557212" Sep 16 18:45:07.158648 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.158592 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6fbqp" podStartSLOduration=1.562371455 podStartE2EDuration="10.158572601s" totalImagesPullingTime="8.596201146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:07.157476642 +0000 UTC m=+44.803571540" watchObservedRunningTime="2026-09-16 18:45:07.158572601 +0000 UTC m=+44.804667409" Sep 16 18:45:07.159132 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.159095 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" podStartSLOduration=12.490986032 podStartE2EDuration="21.159083463s" totalImagesPullingTime="8.668097431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:07.136442422 +0000 UTC m=+44.782537227" watchObservedRunningTime="2026-09-16 18:45:07.159083463 +0000 UTC m=+44.805178260" Sep 16 18:45:07.230415 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:07.230367 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b89554f66-zm44t" podStartSLOduration=12.597837404 podStartE2EDuration="21.230349624s" totalImagesPullingTime="8.63251222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:07.215648797 +0000 UTC m=+44.861743603" watchObservedRunningTime="2026-09-16 18:45:07.230349624 +0000 UTC m=+44.876444430" Sep 16 18:45:08.676819 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:08.676787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:08.822808 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:08.822771 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:09.189883 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:09.189848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:09.300957 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:09.300872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:09.889680 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:09.889645 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:10.012567 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:10.012524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:10.094433 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:10.094396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:12.868783 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:12.868748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:12.952295 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:12.952263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:45:12.952449 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:12.952401 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:12.952449 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:12.952415 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:45:12.952560 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:12.952463 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.952448653 +0000 UTC m=+66.598543437 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:45:13.053384 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:13.053347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:45:13.053578 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:13.053514 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:13.053634 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:13.053580 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:29.053564314 +0000 UTC m=+66.699659098 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:45:22.585316 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:22.585228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:22.707830 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:22.707799 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:22.955799 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:22.955760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:23.028892 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:23.028866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xx2wj" Sep 16 18:45:23.045495 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:23.045457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:23.197628 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:23.197592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:23.426031 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:23.425987 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:26.271690 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:26.271648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:26.353099 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:26.353065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:28.570147 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.570112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:45:28.570572 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.570162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:45:28.572678 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.572658 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:45:28.572744 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.572705 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:45:28.580690 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:28.580670 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:45:28.580785 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:28.580737 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs podName:e3fd2611-8884-4a84-861e-1342cee39e99 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:32.580717693 +0000 UTC m=+130.226812477 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs") pod "network-metrics-daemon-b47hq" (UID: "e3fd2611-8884-4a84-861e-1342cee39e99") : secret "metrics-daemon-secret" not found Sep 16 18:45:28.584177 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.584153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2e9015b-775b-4a7d-8542-9a1222e7f67b-original-pull-secret\") pod \"global-pull-secret-syncer-sm66h\" (UID: \"f2e9015b-775b-4a7d-8542-9a1222e7f67b\") " pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:45:28.594267 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.594248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sm66h" Sep 16 18:45:28.670941 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.670885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:45:28.673795 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.673774 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:45:28.684726 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.684702 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:45:28.695580 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.695557 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtdlx\" (UniqueName: \"kubernetes.io/projected/6b6ef13b-8221-4c89-981c-d17ed779bbca-kube-api-access-dtdlx\") pod \"network-check-target-w8b6s\" (UID: \"6b6ef13b-8221-4c89-981c-d17ed779bbca\") " pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:45:28.892020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.891992 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sm66h"] Sep 16 18:45:28.895215 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:45:28.895190 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e9015b_775b_4a7d_8542_9a1222e7f67b.slice/crio-c26b4764973bf079a548626f5ed8421630ac76f85ed9cc44d87e5630fd781fcf WatchSource:0}: Error finding container c26b4764973bf079a548626f5ed8421630ac76f85ed9cc44d87e5630fd781fcf: Status 404 returned error can't find the container with id c26b4764973bf079a548626f5ed8421630ac76f85ed9cc44d87e5630fd781fcf Sep 16 18:45:28.907805 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.907788 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pf72n\"" Sep 16 18:45:28.915893 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.915877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:45:28.972707 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:28.972681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:45:28.972876 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:28.972851 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:28.972876 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:28.972879 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:45:28.973032 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:28.972947 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:00.972928639 +0000 UTC m=+98.619023426 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:45:29.073567 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:29.073524 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:45:29.073738 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:29.073665 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:29.073738 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:45:29.073724 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:01.0737095 +0000 UTC m=+98.719804287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:45:29.165645 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:29.165558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm66h" event={"ID":"f2e9015b-775b-4a7d-8542-9a1222e7f67b","Type":"ContainerStarted","Data":"c26b4764973bf079a548626f5ed8421630ac76f85ed9cc44d87e5630fd781fcf"} Sep 16 18:45:29.219534 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:29.219501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w8b6s"] Sep 16 18:45:29.222537 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:45:29.222512 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6ef13b_8221_4c89_981c_d17ed779bbca.slice/crio-926e5234ddd85182245eaef417f0156709392888f41a896cefec7c0d2a81a878 WatchSource:0}: Error finding container 926e5234ddd85182245eaef417f0156709392888f41a896cefec7c0d2a81a878: Status 404 returned error can't find the container with id 926e5234ddd85182245eaef417f0156709392888f41a896cefec7c0d2a81a878 Sep 16 18:45:30.170568 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:30.170526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w8b6s" event={"ID":"6b6ef13b-8221-4c89-981c-d17ed779bbca","Type":"ContainerStarted","Data":"926e5234ddd85182245eaef417f0156709392888f41a896cefec7c0d2a81a878"} Sep 16 18:45:30.663361 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:30.663317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:30.726433 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:30.726398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:34.184478 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:34.184441 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w8b6s" event={"ID":"6b6ef13b-8221-4c89-981c-d17ed779bbca","Type":"ContainerStarted","Data":"979823d2f1959da28cd88890cf7572e1d92b170afcd34afbbf7e344ab6b31928"} Sep 16 18:45:34.184969 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:34.184527 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:45:34.186021 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:34.185996 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sm66h" event={"ID":"f2e9015b-775b-4a7d-8542-9a1222e7f67b","Type":"ContainerStarted","Data":"9671ddbd6a8ca8dc5156f6d842cc5214f3e3a964482cc6df6892a53916dca0cf"} Sep 16 18:45:34.200332 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:34.200281 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w8b6s" podStartSLOduration=66.89242304 podStartE2EDuration="1m11.200266921s" totalImagesPullingTime="4.307843881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.1998101 +0000 UTC m=+71.845904907" watchObservedRunningTime="2026-09-16 18:45:34.200266921 +0000 UTC m=+71.846361727" Sep 16 18:45:34.213916 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:34.213877 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sm66h" podStartSLOduration=66.57569221 podStartE2EDuration="1m11.213865358s" totalImagesPullingTime="4.638173148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.213642608 +0000 UTC m=+71.859737415" watchObservedRunningTime="2026-09-16 18:45:34.213865358 +0000 UTC m=+71.859960163" Sep 16 18:45:46.917735 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:46.917608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:48.635219 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:45:48.635190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:00.979898 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:00.979853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:46:00.980280 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:00.980010 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:46:00.980280 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:00.980030 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f9cc85bb7-qdbbt: secret "image-registry-tls" not found Sep 16 18:46:00.980280 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:00.980097 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls podName:f3fc0590-a623-4573-98f6-c668b336b05c nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.980081466 +0000 UTC m=+162.626176250 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls") pod "image-registry-5f9cc85bb7-qdbbt" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c") : secret "image-registry-tls" not found Sep 16 18:46:01.081071 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:01.081038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:46:01.081222 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:01.081136 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:01.081222 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:01.081191 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls podName:77417086-a1e1-4b6a-b8fe-b168e3c07c59 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:05.081177406 +0000 UTC m=+162.727272190 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls") pod "dns-default-8c758" (UID: "77417086-a1e1-4b6a-b8fe-b168e3c07c59") : secret "dns-default-metrics-tls" not found Sep 16 18:46:05.191814 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:05.191785 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w8b6s" Sep 16 18:46:11.900714 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:11.900673 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:12.000967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:12.000930 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:14.061521 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:14.061466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:14.248317 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:14.248278 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:15.861099 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.861066 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8hcq9"] Sep 16 18:46:15.863093 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.863078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:15.865438 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.865418 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 18:46:15.865937 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.865921 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 18:46:15.866012 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.865964 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 18:46:15.866012 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.865974 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 18:46:15.866378 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.866366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6l7cw\"" Sep 16 18:46:15.878533 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.878508 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8hcq9"] Sep 16 18:46:15.891138 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.891106 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:15.962329 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.962288 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:15.975371 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.975339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:15.975526 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.975397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nn6f\" (UniqueName: \"kubernetes.io/projected/4243836b-ee23-4d88-ae22-04619428f84d-kube-api-access-9nn6f\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:15.975526 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.975446 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4243836b-ee23-4d88-ae22-04619428f84d-crio-socket\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:15.975526 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.975495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:15.975647 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:15.975555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4243836b-ee23-4d88-ae22-04619428f84d-data-volume\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.076015 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.075974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4243836b-ee23-4d88-ae22-04619428f84d-data-volume\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.076199 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.076053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.076199 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.076098 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nn6f\" (UniqueName: \"kubernetes.io/projected/4243836b-ee23-4d88-ae22-04619428f84d-kube-api-access-9nn6f\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.076199 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.076124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4243836b-ee23-4d88-ae22-04619428f84d-crio-socket\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.076199 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.076152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.076393 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.076195 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.076393 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.076232 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:16.076393 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.076266 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.576249793 +0000 UTC m=+114.222344577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.076393 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.076301 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.576284958 +0000 UTC m=+114.222379770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : configmap "kube-rbac-proxy" not found Sep 16 18:46:16.076393 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.076226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4243836b-ee23-4d88-ae22-04619428f84d-crio-socket\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.076393 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.076345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4243836b-ee23-4d88-ae22-04619428f84d-data-volume\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.087753 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.087729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nn6f\" (UniqueName: \"kubernetes.io/projected/4243836b-ee23-4d88-ae22-04619428f84d-kube-api-access-9nn6f\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.179590 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.179561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:16.559012 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.558919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:16.580081 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.580058 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.580197 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.580101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:16.580197 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.580168 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:16.580304 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.580217 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.580304 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.580227 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.58021296 +0000 UTC m=+115.226307744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : configmap "kube-rbac-proxy" not found Sep 16 18:46:16.580304 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:16.580298 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.580279714 +0000 UTC m=+115.226374505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:16.660149 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.660120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:16.882520 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.882456 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:16.985880 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:16.985848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:17.085972 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:17.085939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:17.588774 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:17.588741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:17.588948 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:17.588793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:17.588948 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:17.588864 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:17.588948 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:17.588893 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:17.588948 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:17.588934 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.588913335 +0000 UTC m=+117.235008141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : configmap "kube-rbac-proxy" not found Sep 16 18:46:17.588948 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:17.588948 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.588942274 +0000 UTC m=+117.235037058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:18.033577 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:18.033552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4p8g_2e46d10a-d3c7-4b28-bab1-a16ffbe596b1/dns-node-resolver/0.log" Sep 16 18:46:18.483385 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:18.483345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:18.626647 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:18.626615 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:18.638258 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:18.638233 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rcnm2_23954e0d-2798-4c79-8b43-244302141d00/node-ca/0.log" Sep 16 18:46:19.428978 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:19.428951 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6fbqp_735e38d3-3525-407d-9e6c-06aedad02c7a/serve-healthcheck-canary/0.log" Sep 16 18:46:19.604716 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:19.604686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:19.604863 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:19.604734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:19.604863 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:19.604830 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:19.604863 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:19.604841 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:19.604963 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:19.604895 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.604879343 +0000 UTC m=+121.250974325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : configmap "kube-rbac-proxy" not found Sep 16 18:46:19.604963 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:19.604910 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.604904719 +0000 UTC m=+121.250999503 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:19.852255 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:19.852177 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:19.957433 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:19.957400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:20.281787 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:20.281751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:20.417091 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:20.417061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:20.613944 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:20.613855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:20.733604 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:20.733571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:21.614053 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:21.614021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:21.947520 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:21.947469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:22.171874 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:22.171841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:22.306233 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:22.306146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:22.504249 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:22.504219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:23.116401 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:23.116363 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:23.482531 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:23.482473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:23.639458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:23.639418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:23.639635 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:23.639474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:23.639635 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:23.639607 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:46:23.639712 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:23.639607 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:46:23.639748 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:23.639682 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:31.6396659 +0000 UTC m=+129.285760684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : secret "insights-runtime-extractor-tls" not found Sep 16 18:46:23.639748 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:46:23.639745 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm podName:4243836b-ee23-4d88-ae22-04619428f84d nodeName:}" failed. No retries permitted until 2026-09-16 18:46:31.639732863 +0000 UTC m=+129.285827647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hcq9" (UID: "4243836b-ee23-4d88-ae22-04619428f84d") : configmap "kube-rbac-proxy" not found Sep 16 18:46:23.891756 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:23.891717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:24.142927 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:24.142829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:24.329254 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:24.329216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:24.466201 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:24.466108 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:24.568100 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:24.568070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:25.734585 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:25.734552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:25.891863 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:25.891838 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:26.075876 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:26.075797 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:26.163317 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:26.163285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:26.289023 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:26.288991 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:26.496554 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:26.496521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:26.597255 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:26.597220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:27.475845 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:27.475800 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" podUID="3ac554d4-1b23-4222-82e9-129d22e14e53" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 18:46:27.476875 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:27.476855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:27.882698 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:27.882662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:28.076259 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:28.076224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:28.278886 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:28.278804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:28.476703 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:28.476667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:29.080303 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:29.080274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:29.477306 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:29.477269 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:29.870089 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:29.870014 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:29.968543 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:29.968507 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:30.067288 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:30.067260 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:30.167641 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:30.167610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:30.295409 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:30.295364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:30.482874 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:30.482785 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:31.104536 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.104504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:31.322683 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.322643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:31.500127 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.500092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:31.704904 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.704867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:31.705086 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.704917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:31.705361 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.705346 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4243836b-ee23-4d88-ae22-04619428f84d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:31.707249 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.707230 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4243836b-ee23-4d88-ae22-04619428f84d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hcq9\" (UID: \"4243836b-ee23-4d88-ae22-04619428f84d\") " pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:31.774429 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.774361 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6l7cw\"" Sep 16 18:46:31.782343 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:31.782327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8hcq9" Sep 16 18:46:32.073154 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.073132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8hcq9"] Sep 16 18:46:32.075748 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:46:32.075721 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4243836b_ee23_4d88_ae22_04619428f84d.slice/crio-e510c6118df965ef7db070463e0f761f27ef5fb302bf82f82a3a06e82e7ede6b WatchSource:0}: Error finding container e510c6118df965ef7db070463e0f761f27ef5fb302bf82f82a3a06e82e7ede6b: Status 404 returned error can't find the container with id e510c6118df965ef7db070463e0f761f27ef5fb302bf82f82a3a06e82e7ede6b Sep 16 18:46:32.357450 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.357356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hcq9" event={"ID":"4243836b-ee23-4d88-ae22-04619428f84d","Type":"ContainerStarted","Data":"2b4ebbf3f9a28e1b3c126909f8ed2f9c60dfdd6728406be4e7a922d06cf45e51"} Sep 16 18:46:32.357450 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.357420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hcq9" event={"ID":"4243836b-ee23-4d88-ae22-04619428f84d","Type":"ContainerStarted","Data":"e510c6118df965ef7db070463e0f761f27ef5fb302bf82f82a3a06e82e7ede6b"} Sep 16 18:46:32.612349 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.612246 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:46:32.614750 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.614729 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3fd2611-8884-4a84-861e-1342cee39e99-metrics-certs\") pod \"network-metrics-daemon-b47hq\" (UID: \"e3fd2611-8884-4a84-861e-1342cee39e99\") " pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:46:32.815469 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.815436 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dgmmm\"" Sep 16 18:46:32.823317 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.823296 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b47hq" Sep 16 18:46:32.900211 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.900180 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:32.999403 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:32.999364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:33.120152 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:33.120126 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b47hq"] Sep 16 18:46:33.122939 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:46:33.122916 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fd2611_8884_4a84_861e_1342cee39e99.slice/crio-84ab4d87d707a42c8328101234842f4643d44c7d53329b8041a86913e4e6701e WatchSource:0}: Error finding container 84ab4d87d707a42c8328101234842f4643d44c7d53329b8041a86913e4e6701e: Status 404 returned error can't find the container with id 84ab4d87d707a42c8328101234842f4643d44c7d53329b8041a86913e4e6701e Sep 16 18:46:33.361610 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:33.361578 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hcq9" event={"ID":"4243836b-ee23-4d88-ae22-04619428f84d","Type":"ContainerStarted","Data":"862c5653236106d2a1e10b52ab366ef4059488be5ef8a894becd237880a6ceaa"} Sep 16 18:46:33.362649 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:33.362625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b47hq" event={"ID":"e3fd2611-8884-4a84-861e-1342cee39e99","Type":"ContainerStarted","Data":"84ab4d87d707a42c8328101234842f4643d44c7d53329b8041a86913e4e6701e"} Sep 16 18:46:35.040385 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.040347 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:35.142149 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.142102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:35.318146 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.318058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:35.370527 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.370477 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hcq9" event={"ID":"4243836b-ee23-4d88-ae22-04619428f84d","Type":"ContainerStarted","Data":"5632231bab9af9d2028c3ef686bc6c61d1d9701d6b9a9b2046df8088cbc2f039"} Sep 16 18:46:35.372138 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.372112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b47hq" event={"ID":"e3fd2611-8884-4a84-861e-1342cee39e99","Type":"ContainerStarted","Data":"b6bbab231c427d02dae6c0f3f7b042cacdcea87cf92986e5512d5855cb1248e4"} Sep 16 18:46:35.372236 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.372159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b47hq" event={"ID":"e3fd2611-8884-4a84-861e-1342cee39e99","Type":"ContainerStarted","Data":"089919f60452d9c13a5f28d3374f90413a20fb56e8fb2d9ed3e10c150fb2fde9"} Sep 16 18:46:35.393259 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.393209 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8hcq9" podStartSLOduration=18.204525558 podStartE2EDuration="20.393194611s" totalImagesPullingTime="2.188669053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:35.392466896 +0000 UTC m=+133.038561693" watchObservedRunningTime="2026-09-16 18:46:35.393194611 +0000 UTC m=+133.039289415" Sep 16 18:46:35.414685 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.414641 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b47hq" podStartSLOduration=131.149169171 podStartE2EDuration="2m12.414628392s" totalImagesPullingTime="1.265459221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:35.413971376 +0000 UTC m=+133.060066185" watchObservedRunningTime="2026-09-16 18:46:35.414628392 +0000 UTC m=+133.060723197" Sep 16 18:46:35.695240 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.695200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:35.753787 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:35.753761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:36.094831 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:36.094748 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:36.170019 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:36.169984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:36.306083 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:36.306051 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:36.371747 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:36.371669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:36.716400 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:36.716364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:36.907948 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:36.907910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:37.008493 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:37.008393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:37.476325 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:37.476286 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" podUID="3ac554d4-1b23-4222-82e9-129d22e14e53" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 18:46:38.342316 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:38.342285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:38.426668 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:38.426623 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:43.603010 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:43.602966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:43.756019 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:43.755986 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:43.870515 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:43.870430 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:44.052239 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:44.052201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:44.147318 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:44.147287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:45.526056 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:45.526023 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:45.589735 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:45.589707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:46.089436 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:46.089398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:46.191146 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:46.191113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:46.305085 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:46.305054 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:47.476156 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:47.476116 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" podUID="3ac554d4-1b23-4222-82e9-129d22e14e53" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 18:46:47.476638 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:47.476188 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" Sep 16 18:46:47.476800 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:47.476767 2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c2b3f10d316b3198edc6c637a24360e9f31eae9ca2e69b49b20811932a40b96e"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 16 18:46:47.476866 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:47.476826 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" podUID="3ac554d4-1b23-4222-82e9-129d22e14e53" containerName="service-proxy" containerID="cri-o://c2b3f10d316b3198edc6c637a24360e9f31eae9ca2e69b49b20811932a40b96e" gracePeriod=30 Sep 16 18:46:48.415166 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:48.415134 2576 generic.go:356] "Generic (PLEG): container finished" podID="3ac554d4-1b23-4222-82e9-129d22e14e53" containerID="c2b3f10d316b3198edc6c637a24360e9f31eae9ca2e69b49b20811932a40b96e" exitCode=2 Sep 16 18:46:48.415334 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:48.415183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" event={"ID":"3ac554d4-1b23-4222-82e9-129d22e14e53","Type":"ContainerDied","Data":"c2b3f10d316b3198edc6c637a24360e9f31eae9ca2e69b49b20811932a40b96e"} Sep 16 18:46:48.415334 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:48.415219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9b4695f46-4nszf" event={"ID":"3ac554d4-1b23-4222-82e9-129d22e14e53","Type":"ContainerStarted","Data":"22650e8567952b128f286dc91255ebca8b535d6cdfc0c32114efbc1c885e690a"} Sep 16 18:46:48.471642 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:48.471605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:50.304313 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.304280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:50.421795 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.421750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:50.530537 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.530503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:50.656034 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.656007 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7j7vb"] Sep 16 18:46:50.659189 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.659168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.663203 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.663177 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 18:46:50.663350 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.663204 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 18:46:50.663350 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.663177 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 18:46:50.663672 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.663650 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 18:46:50.663781 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.663676 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-22rpb\"" Sep 16 18:46:50.664180 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.664166 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 18:46:50.664225 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.664197 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 18:46:50.686534 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.686501 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:50.755222 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755232 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-root\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755253 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-wtmp\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49vf\" (UniqueName: \"kubernetes.io/projected/a8dd90d6-b781-42af-8aa8-2812cd343c88-kube-api-access-m49vf\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8dd90d6-b781-42af-8aa8-2812cd343c88-metrics-client-ca\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-sys\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755383 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755376 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-textfile\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755601 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-tls\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.755601 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.755431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.804292 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.804262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:50.856398 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-root\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-wtmp\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49vf\" (UniqueName: \"kubernetes.io/projected/a8dd90d6-b781-42af-8aa8-2812cd343c88-kube-api-access-m49vf\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8dd90d6-b781-42af-8aa8-2812cd343c88-metrics-client-ca\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856475 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-sys\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-root\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856551 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856555 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-sys\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856920 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-wtmp\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856920 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-textfile\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856920 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-tls\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856920 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.856920 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.856917 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-textfile\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.857175 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.857043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-accelerators-collector-config\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.857175 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.857139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8dd90d6-b781-42af-8aa8-2812cd343c88-metrics-client-ca\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.859166 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.859139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-tls\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.859166 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.859162 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8dd90d6-b781-42af-8aa8-2812cd343c88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.868244 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.868224 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49vf\" (UniqueName: \"kubernetes.io/projected/a8dd90d6-b781-42af-8aa8-2812cd343c88-kube-api-access-m49vf\") pod \"node-exporter-7j7vb\" (UID: \"a8dd90d6-b781-42af-8aa8-2812cd343c88\") " pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.908028 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.907957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:50.968069 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:50.968023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7j7vb" Sep 16 18:46:50.979074 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:46:50.979043 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dd90d6_b781_42af_8aa8_2812cd343c88.slice/crio-9d7afa7b2f08d74e55fd267e820f6b343795ed20a6372ce291e27caff3a07b67 WatchSource:0}: Error finding container 9d7afa7b2f08d74e55fd267e820f6b343795ed20a6372ce291e27caff3a07b67: Status 404 returned error can't find the container with id 9d7afa7b2f08d74e55fd267e820f6b343795ed20a6372ce291e27caff3a07b67 Sep 16 18:46:51.008255 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.008220 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:51.101175 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.101143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:51.202570 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.202471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:51.389543 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.389510 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:51.426068 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.426043 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j7vb" event={"ID":"a8dd90d6-b781-42af-8aa8-2812cd343c88","Type":"ContainerStarted","Data":"9d7afa7b2f08d74e55fd267e820f6b343795ed20a6372ce291e27caff3a07b67"} Sep 16 18:46:51.611892 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.611809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:51.696010 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.695973 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:51.818106 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:51.818074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:52.219682 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:52.219653 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:52.431023 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:52.430994 2576 generic.go:356] "Generic (PLEG): container finished" podID="a8dd90d6-b781-42af-8aa8-2812cd343c88" containerID="80aaca9e6a3ca1a17cf12df5378d4c27b8aaea3b5f7dc430a472b28dd896fa98" exitCode=0 Sep 16 18:46:52.431379 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:52.431060 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j7vb" event={"ID":"a8dd90d6-b781-42af-8aa8-2812cd343c88","Type":"ContainerDied","Data":"80aaca9e6a3ca1a17cf12df5378d4c27b8aaea3b5f7dc430a472b28dd896fa98"} Sep 16 18:46:52.456589 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:52.456545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:52.527040 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:52.527009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:53.436278 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:53.436229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j7vb" event={"ID":"a8dd90d6-b781-42af-8aa8-2812cd343c88","Type":"ContainerStarted","Data":"ee9a46a40e98940859ff5de03f83e1321a85f94b4ee35e7920c0dd966c397887"} Sep 16 18:46:53.436705 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:53.436289 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7j7vb" event={"ID":"a8dd90d6-b781-42af-8aa8-2812cd343c88","Type":"ContainerStarted","Data":"47f527d818a5ae5a7e3bbc2bd3bd7033189ef60ae3c926328c19ff57496f6ec3"} Sep 16 18:46:53.457616 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:53.457561 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7j7vb" podStartSLOduration=2.328149573 podStartE2EDuration="3.457546406s" totalImagesPullingTime="1.059718005s" totalInitContainerRuntime="69.678828ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:53.456693301 +0000 UTC m=+151.102788106" watchObservedRunningTime="2026-09-16 18:46:53.457546406 +0000 UTC m=+151.103641211" Sep 16 18:46:53.482604 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:53.482569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:53.589223 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:53.589198 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:53.679683 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:53.679643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:54.081747 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:54.081716 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:54.182919 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:54.182888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:55.330717 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:55.330679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:55.428831 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:55.428796 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:55.739583 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:55.739552 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:56.340686 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:56.340638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:56.402500 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:56.402453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:56.804600 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:56.804569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:59.355973 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:59.355943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:59.456615 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:46:59.456584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:00.150210 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:47:00.150170 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" podUID="f3fc0590-a623-4573-98f6-c668b336b05c" Sep 16 18:47:00.196543 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:47:00.196505 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8c758" podUID="77417086-a1e1-4b6a-b8fe-b168e3c07c59" Sep 16 18:47:00.318390 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:00.318358 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:00.421613 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:00.421515 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:00.461846 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:00.461815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:00.461996 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:00.461815 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8c758" Sep 16 18:47:02.882007 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:02.881970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:02.986079 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:02.986042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:03.245219 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:03.245096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:03.348574 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:03.348543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:05.055459 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.055424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:05.057971 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.057943 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"image-registry-5f9cc85bb7-qdbbt\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:05.156733 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.156697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:47:05.159049 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.159031 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77417086-a1e1-4b6a-b8fe-b168e3c07c59-metrics-tls\") pod \"dns-default-8c758\" (UID: \"77417086-a1e1-4b6a-b8fe-b168e3c07c59\") " pod="openshift-dns/dns-default-8c758" Sep 16 18:47:05.264543 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.264509 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g84ch\"" Sep 16 18:47:05.265064 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.265049 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7kp4l\"" Sep 16 18:47:05.272854 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.272833 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8c758" Sep 16 18:47:05.272953 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.272936 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:05.553331 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.553298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8c758"] Sep 16 18:47:05.556788 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.556762 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f9cc85bb7-qdbbt"] Sep 16 18:47:05.556788 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:47:05.556778 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77417086_a1e1_4b6a_b8fe_b168e3c07c59.slice/crio-9b32782f1c2dda9a4f8bdea2028ad51bb038c7ed0289fc1793443f10df0b139c WatchSource:0}: Error finding container 9b32782f1c2dda9a4f8bdea2028ad51bb038c7ed0289fc1793443f10df0b139c: Status 404 returned error can't find the container with id 9b32782f1c2dda9a4f8bdea2028ad51bb038c7ed0289fc1793443f10df0b139c Sep 16 18:47:05.560668 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:47:05.560644 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fc0590_a623_4573_98f6_c668b336b05c.slice/crio-42ddf4115137c8b841349b604c8b7a09349cef634217609e1476277bbd2e69ff WatchSource:0}: Error finding container 42ddf4115137c8b841349b604c8b7a09349cef634217609e1476277bbd2e69ff: Status 404 returned error can't find the container with id 42ddf4115137c8b841349b604c8b7a09349cef634217609e1476277bbd2e69ff Sep 16 18:47:05.882752 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.882715 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f9cc85bb7-qdbbt"] Sep 16 18:47:05.980438 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:05.980407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:06.081019 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:06.080984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:06.482744 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:06.482711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8c758" event={"ID":"77417086-a1e1-4b6a-b8fe-b168e3c07c59","Type":"ContainerStarted","Data":"9b32782f1c2dda9a4f8bdea2028ad51bb038c7ed0289fc1793443f10df0b139c"} Sep 16 18:47:06.483997 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:06.483976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" event={"ID":"f3fc0590-a623-4573-98f6-c668b336b05c","Type":"ContainerStarted","Data":"dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539"} Sep 16 18:47:06.484119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:06.484018 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" event={"ID":"f3fc0590-a623-4573-98f6-c668b336b05c","Type":"ContainerStarted","Data":"42ddf4115137c8b841349b604c8b7a09349cef634217609e1476277bbd2e69ff"} Sep 16 18:47:06.484119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:06.484106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:06.509874 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:06.509823 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" podStartSLOduration=163.509807215 podStartE2EDuration="2m43.509807215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:06.50804231 +0000 UTC m=+164.154137117" watchObservedRunningTime="2026-09-16 18:47:06.509807215 +0000 UTC m=+164.155902021" Sep 16 18:47:07.489548 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:07.489507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8c758" event={"ID":"77417086-a1e1-4b6a-b8fe-b168e3c07c59","Type":"ContainerStarted","Data":"9d008409032dcd77603d783d6bd30527af87bb9c15430e092d7d1c796511207b"} Sep 16 18:47:07.489986 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:07.489568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8c758" event={"ID":"77417086-a1e1-4b6a-b8fe-b168e3c07c59","Type":"ContainerStarted","Data":"c955305fe0da2409e6733e409653ea2e23bb42ecf19f500a9beac1ebb1607525"} Sep 16 18:47:07.562124 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:07.562062 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8c758" podStartSLOduration=129.024110217 podStartE2EDuration="2m10.562042265s" totalImagesPullingTime="1.537932048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:07.561755787 +0000 UTC m=+165.207850593" watchObservedRunningTime="2026-09-16 18:47:07.562042265 +0000 UTC m=+165.208137073" Sep 16 18:47:08.492929 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:08.492900 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8c758" Sep 16 18:47:09.739317 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:09.739287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:09.840140 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:09.840109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:11.398700 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:11.398669 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:11.549133 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:11.549093 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:11.630241 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:11.630204 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:17.872454 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:17.872365 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:17.972741 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:17.972702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:18.497732 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:18.497706 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8c758" Sep 16 18:47:20.245822 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:20.245783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:20.343623 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:20.343593 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:20.409288 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:20.409252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:24.183529 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:24.183469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:24.296680 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:24.296652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:27.496466 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:27.496433 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:27.651596 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:27.651561 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:27.720795 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:27.720750 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:32.508106 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.508049 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" podUID="f3fc0590-a623-4573-98f6-c668b336b05c" containerName="registry" containerID="cri-o://dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539" gracePeriod=30 Sep 16 18:47:32.741975 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.741953 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:32.846072 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846000 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-bound-sa-token\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846072 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846034 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-image-registry-private-configuration\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846072 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846054 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3fc0590-a623-4573-98f6-c668b336b05c-ca-trust-extracted\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846315 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846097 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-installation-pull-secrets\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846315 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846315 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846163 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r799c\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-kube-api-access-r799c\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846315 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846222 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-registry-certificates\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846315 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-trusted-ca\") pod \"f3fc0590-a623-4573-98f6-c668b336b05c\" (UID: \"f3fc0590-a623-4573-98f6-c668b336b05c\") " Sep 16 18:47:32.846776 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:32.846982 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.846958 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:47:32.848735 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.848702 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:47:32.848934 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.848910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:47:32.848999 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.848961 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:47:32.849126 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.849104 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-kube-api-access-r799c" (OuterVolumeSpecName: "kube-api-access-r799c") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "kube-api-access-r799c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:47:32.849209 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.849189 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:47:32.855018 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.854993 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3fc0590-a623-4573-98f6-c668b336b05c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f3fc0590-a623-4573-98f6-c668b336b05c" (UID: "f3fc0590-a623-4573-98f6-c668b336b05c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:47:32.947089 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947062 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-registry-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.947089 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947086 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r799c\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-kube-api-access-r799c\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.947221 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947097 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-registry-certificates\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.947221 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947111 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f3fc0590-a623-4573-98f6-c668b336b05c-trusted-ca\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.947221 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947120 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f3fc0590-a623-4573-98f6-c668b336b05c-bound-sa-token\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.947221 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947129 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-image-registry-private-configuration\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.947221 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947137 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f3fc0590-a623-4573-98f6-c668b336b05c-ca-trust-extracted\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:32.947221 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:32.947145 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f3fc0590-a623-4573-98f6-c668b336b05c-installation-pull-secrets\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:47:33.572698 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.572667 2576 generic.go:356] "Generic (PLEG): container finished" podID="f3fc0590-a623-4573-98f6-c668b336b05c" containerID="dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539" exitCode=0 Sep 16 18:47:33.573167 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.572721 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" Sep 16 18:47:33.573167 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.572736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" event={"ID":"f3fc0590-a623-4573-98f6-c668b336b05c","Type":"ContainerDied","Data":"dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539"} Sep 16 18:47:33.573167 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.572803 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f9cc85bb7-qdbbt" event={"ID":"f3fc0590-a623-4573-98f6-c668b336b05c","Type":"ContainerDied","Data":"42ddf4115137c8b841349b604c8b7a09349cef634217609e1476277bbd2e69ff"} Sep 16 18:47:33.573167 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.572821 2576 scope.go:117] "RemoveContainer" containerID="dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539" Sep 16 18:47:33.582735 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.582711 2576 scope.go:117] "RemoveContainer" containerID="dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539" Sep 16 18:47:33.583082 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:47:33.583061 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539\": container with ID starting with dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539 not found: ID does not exist" containerID="dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539" Sep 16 18:47:33.583145 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.583092 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539"} err="failed to get container status \"dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539\": rpc error: code = NotFound desc = could not find container \"dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539\": container with ID starting with dfa16675d3d7e349860a396c174f7d6b6bd3f265e8751db975699e51646ab539 not found: ID does not exist" Sep 16 18:47:33.602312 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.602280 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f9cc85bb7-qdbbt"] Sep 16 18:47:33.620136 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:33.620111 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f9cc85bb7-qdbbt"] Sep 16 18:47:34.881972 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:34.881932 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3fc0590-a623-4573-98f6-c668b336b05c" path="/var/lib/kubelet/pods/f3fc0590-a623-4573-98f6-c668b336b05c/volumes" Sep 16 18:47:36.657069 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:36.657037 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:36.724899 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:47:36.724857 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:06.789864 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:06.789827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:06.954846 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:06.954809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:18.763806 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:18.763740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:18.840305 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:18.840272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:28.142056 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:28.142021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:32.889697 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:32.889660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:32.992238 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:32.992213 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:35.145216 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:35.145181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:35.232879 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:35.232841 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:45.674161 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:45.674079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:45.775016 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:48:45.774978 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:22.770701 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:22.770662 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:49:22.775122 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:22.773624 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 18:49:22.781865 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:22.781844 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:49:30.165245 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:30.165206 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:30.262791 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:30.262760 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:30.327115 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:30.327077 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:53.462286 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:53.462246 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:53.558989 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:53.558955 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:58.887938 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:58.887898 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:59.159925 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.159847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:59.600338 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.600254 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:59.697074 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.697028 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:59.800777 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.800751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:59.810573 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.810552 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp"] Sep 16 18:49:59.810796 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.810785 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3fc0590-a623-4573-98f6-c668b336b05c" containerName="registry" Sep 16 18:49:59.810841 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.810798 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3fc0590-a623-4573-98f6-c668b336b05c" containerName="registry" Sep 16 18:49:59.810874 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.810844 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3fc0590-a623-4573-98f6-c668b336b05c" containerName="registry" Sep 16 18:49:59.813451 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.813438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:49:59.817341 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.817323 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 18:49:59.818373 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.818238 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 16 18:49:59.818373 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.818250 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-d85hl\"" Sep 16 18:49:59.818373 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.818270 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 18:49:59.818373 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.818303 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 18:49:59.818695 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.818576 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 18:49:59.825529 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.825508 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp"] Sep 16 18:49:59.936618 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.936591 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp666\" (UniqueName: \"kubernetes.io/projected/80aab95f-2af3-41eb-a7d1-025032b12d13-kube-api-access-zp666\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:49:59.936618 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.936622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/80aab95f-2af3-41eb-a7d1-025032b12d13-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:49:59.937020 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.936718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/80aab95f-2af3-41eb-a7d1-025032b12d13-certificates\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:49:59.963894 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.963865 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-gvn5c"] Sep 16 18:49:59.966766 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.966752 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:49:59.969988 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.969970 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 16 18:49:59.974967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.974942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:59.979777 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:49:59.979755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-gvn5c"] Sep 16 18:50:00.037499 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.037447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lf2v\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-kube-api-access-5lf2v\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:00.037668 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.037509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/80aab95f-2af3-41eb-a7d1-025032b12d13-certificates\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:00.037668 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.037538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:00.037668 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.037576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp666\" (UniqueName: \"kubernetes.io/projected/80aab95f-2af3-41eb-a7d1-025032b12d13-kube-api-access-zp666\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:00.037668 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.037612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/80aab95f-2af3-41eb-a7d1-025032b12d13-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:00.038000 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.037983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/80aab95f-2af3-41eb-a7d1-025032b12d13-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:00.040042 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.040026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/80aab95f-2af3-41eb-a7d1-025032b12d13-certificates\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:00.046837 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.046811 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp666\" (UniqueName: \"kubernetes.io/projected/80aab95f-2af3-41eb-a7d1-025032b12d13-kube-api-access-zp666\") pod \"keda-metrics-apiserver-77857b7b98-qx8hp\" (UID: \"80aab95f-2af3-41eb-a7d1-025032b12d13\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:00.070911 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.070880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:00.123786 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.123755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:00.138917 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.138886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lf2v\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-kube-api-access-5lf2v\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:00.139011 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.138942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:00.139101 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:50:00.139081 2576 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Sep 16 18:50:00.139222 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:50:00.139109 2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-gvn5c: secret "keda-admission-webhooks-certs" not found Sep 16 18:50:00.139222 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:50:00.139176 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates podName:0acac2aa-d55d-4c46-bd20-66bdf14e35ed nodeName:}" failed. No retries permitted until 2026-09-16 18:50:00.639154096 +0000 UTC m=+338.285248892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates") pod "keda-admission-6c4d864b86-gvn5c" (UID: "0acac2aa-d55d-4c46-bd20-66bdf14e35ed") : secret "keda-admission-webhooks-certs" not found Sep 16 18:50:00.157874 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.157850 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lf2v\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-kube-api-access-5lf2v\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:00.397206 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.397173 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp"] Sep 16 18:50:00.400126 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:50:00.400102 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80aab95f_2af3_41eb_a7d1_025032b12d13.slice/crio-8bf61bf802e1f12d0ab079b94b70f0176d69e496e1827616a830853c5d4c5fb3 WatchSource:0}: Error finding container 8bf61bf802e1f12d0ab079b94b70f0176d69e496e1827616a830853c5d4c5fb3: Status 404 returned error can't find the container with id 8bf61bf802e1f12d0ab079b94b70f0176d69e496e1827616a830853c5d4c5fb3 Sep 16 18:50:00.401348 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.401329 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:50:00.433094 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.433065 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:00.642209 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.642180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:00.642351 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:50:00.642305 2576 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found Sep 16 18:50:00.642351 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:50:00.642322 2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-gvn5c: secret "keda-admission-webhooks-certs" not found Sep 16 18:50:00.642419 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:50:00.642372 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates podName:0acac2aa-d55d-4c46-bd20-66bdf14e35ed nodeName:}" failed. No retries permitted until 2026-09-16 18:50:01.642357082 +0000 UTC m=+339.288451866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates") pod "keda-admission-6c4d864b86-gvn5c" (UID: "0acac2aa-d55d-4c46-bd20-66bdf14e35ed") : secret "keda-admission-webhooks-certs" not found Sep 16 18:50:00.686614 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.686535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:00.793389 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.793350 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:00.939666 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:00.939584 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:01.031885 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:01.031857 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" event={"ID":"80aab95f-2af3-41eb-a7d1-025032b12d13","Type":"ContainerStarted","Data":"8bf61bf802e1f12d0ab079b94b70f0176d69e496e1827616a830853c5d4c5fb3"} Sep 16 18:50:01.649859 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:01.649817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:01.652702 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:01.652679 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0acac2aa-d55d-4c46-bd20-66bdf14e35ed-certificates\") pod \"keda-admission-6c4d864b86-gvn5c\" (UID: \"0acac2aa-d55d-4c46-bd20-66bdf14e35ed\") " pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:01.775994 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:01.775959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:02.099334 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:02.099259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-gvn5c"] Sep 16 18:50:03.553305 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:50:03.553270 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acac2aa_d55d_4c46_bd20_66bdf14e35ed.slice/crio-dfd020d465cdae9311f5e52978c23a6a3ae125a35f60985b6f75c1cc6d4e6021 WatchSource:0}: Error finding container dfd020d465cdae9311f5e52978c23a6a3ae125a35f60985b6f75c1cc6d4e6021: Status 404 returned error can't find the container with id dfd020d465cdae9311f5e52978c23a6a3ae125a35f60985b6f75c1cc6d4e6021 Sep 16 18:50:04.044573 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:04.044526 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" event={"ID":"0acac2aa-d55d-4c46-bd20-66bdf14e35ed","Type":"ContainerStarted","Data":"dfd020d465cdae9311f5e52978c23a6a3ae125a35f60985b6f75c1cc6d4e6021"} Sep 16 18:50:04.046197 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:04.046165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" event={"ID":"80aab95f-2af3-41eb-a7d1-025032b12d13","Type":"ContainerStarted","Data":"090aca4aed1a3e2a088a58e3745b329c70b36d7e482dd7114cd3684a71b436ca"} Sep 16 18:50:04.046332 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:04.046316 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:04.082593 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:04.082539 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" podStartSLOduration=1.86893189 podStartE2EDuration="5.082523707s" totalImagesPullingTime="3.213591817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:04.081644101 +0000 UTC m=+341.727738908" watchObservedRunningTime="2026-09-16 18:50:04.082523707 +0000 UTC m=+341.728618512" Sep 16 18:50:07.059421 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:07.059388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" event={"ID":"0acac2aa-d55d-4c46-bd20-66bdf14e35ed","Type":"ContainerStarted","Data":"97fa093aad91da1c88aa519e1989e6068f2b8bf89a95fd47fc6187630630e727"} Sep 16 18:50:07.059833 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:07.059538 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:07.091288 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:07.091233 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" podStartSLOduration=5.361525182 podStartE2EDuration="8.091216374s" totalImagesPullingTime="2.729691192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:07.090391256 +0000 UTC m=+344.736486062" watchObservedRunningTime="2026-09-16 18:50:07.091216374 +0000 UTC m=+344.737311177" Sep 16 18:50:07.197744 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:07.197711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:07.317812 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:07.317726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:15.056017 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:15.055987 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-qx8hp" Sep 16 18:50:28.065308 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:28.065272 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-gvn5c" Sep 16 18:50:51.929999 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:51.929959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:53.323524 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:50:53.323455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:03.770400 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:03.770361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:03.901102 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:03.901066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:19.390404 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.390360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:19.449176 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.449146 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-lptb5"] Sep 16 18:51:19.452273 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.452252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.454676 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.454654 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-2l4cs\"" Sep 16 18:51:19.454768 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.454749 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 18:51:19.455318 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.455302 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 16 18:51:19.455372 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.455361 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 18:51:19.462468 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.462447 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-lptb5"] Sep 16 18:51:19.483177 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.483155 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-bcld6"] Sep 16 18:51:19.486074 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.486056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.487098 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.487067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:19.490017 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.490001 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 18:51:19.490099 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.490043 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-6x7qm\"" Sep 16 18:51:19.495160 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.495139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bcld6"] Sep 16 18:51:19.500292 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.500266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc2e503-0557-4530-b71f-ddeff192e2e3-cert\") pod \"kserve-controller-manager-f8c46b677-lptb5\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.500387 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.500313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d82gc\" (UniqueName: \"kubernetes.io/projected/7fc2e503-0557-4530-b71f-ddeff192e2e3-kube-api-access-d82gc\") pod \"kserve-controller-manager-f8c46b677-lptb5\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.553894 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.553869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:19.601537 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.601512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc2e503-0557-4530-b71f-ddeff192e2e3-cert\") pod \"kserve-controller-manager-f8c46b677-lptb5\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.601657 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.601551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzf9x\" (UniqueName: \"kubernetes.io/projected/36905445-43b6-4adb-8fc0-d754b3aa6a63-kube-api-access-kzf9x\") pod \"seaweedfs-64568bcd49-bcld6\" (UID: \"36905445-43b6-4adb-8fc0-d754b3aa6a63\") " pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.601657 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.601584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d82gc\" (UniqueName: \"kubernetes.io/projected/7fc2e503-0557-4530-b71f-ddeff192e2e3-kube-api-access-d82gc\") pod \"kserve-controller-manager-f8c46b677-lptb5\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.601657 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.601603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36905445-43b6-4adb-8fc0-d754b3aa6a63-data\") pod \"seaweedfs-64568bcd49-bcld6\" (UID: \"36905445-43b6-4adb-8fc0-d754b3aa6a63\") " pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.604060 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.604041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc2e503-0557-4530-b71f-ddeff192e2e3-cert\") pod \"kserve-controller-manager-f8c46b677-lptb5\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.609954 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.609933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d82gc\" (UniqueName: \"kubernetes.io/projected/7fc2e503-0557-4530-b71f-ddeff192e2e3-kube-api-access-d82gc\") pod \"kserve-controller-manager-f8c46b677-lptb5\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.653678 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.653621 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:19.702908 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.702885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzf9x\" (UniqueName: \"kubernetes.io/projected/36905445-43b6-4adb-8fc0-d754b3aa6a63-kube-api-access-kzf9x\") pod \"seaweedfs-64568bcd49-bcld6\" (UID: \"36905445-43b6-4adb-8fc0-d754b3aa6a63\") " pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.702996 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.702915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36905445-43b6-4adb-8fc0-d754b3aa6a63-data\") pod \"seaweedfs-64568bcd49-bcld6\" (UID: \"36905445-43b6-4adb-8fc0-d754b3aa6a63\") " pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.703215 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.703200 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36905445-43b6-4adb-8fc0-d754b3aa6a63-data\") pod \"seaweedfs-64568bcd49-bcld6\" (UID: \"36905445-43b6-4adb-8fc0-d754b3aa6a63\") " pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.712388 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.712361 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzf9x\" (UniqueName: \"kubernetes.io/projected/36905445-43b6-4adb-8fc0-d754b3aa6a63-kube-api-access-kzf9x\") pod \"seaweedfs-64568bcd49-bcld6\" (UID: \"36905445-43b6-4adb-8fc0-d754b3aa6a63\") " pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.762938 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.762919 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:19.795035 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.795011 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:19.955454 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:19.955366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:20.039360 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:20.039334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-lptb5"] Sep 16 18:51:20.042202 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:51:20.042180 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc2e503_0557_4530_b71f_ddeff192e2e3.slice/crio-2b97fe35c1405aa12f5ee480e415ae644a870c736c89362588e98ddfe5d6811e WatchSource:0}: Error finding container 2b97fe35c1405aa12f5ee480e415ae644a870c736c89362588e98ddfe5d6811e: Status 404 returned error can't find the container with id 2b97fe35c1405aa12f5ee480e415ae644a870c736c89362588e98ddfe5d6811e Sep 16 18:51:20.080068 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:20.080036 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bcld6"] Sep 16 18:51:20.082932 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:51:20.082905 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36905445_43b6_4adb_8fc0_d754b3aa6a63.slice/crio-7ebb3d9c6d692a802f5352bf935bb3c55698795b85072351a80bf0427dbea6a7 WatchSource:0}: Error finding container 7ebb3d9c6d692a802f5352bf935bb3c55698795b85072351a80bf0427dbea6a7: Status 404 returned error can't find the container with id 7ebb3d9c6d692a802f5352bf935bb3c55698795b85072351a80bf0427dbea6a7 Sep 16 18:51:20.131209 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:20.131179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:20.260564 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:20.260462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:20.314117 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:20.314081 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bcld6" event={"ID":"36905445-43b6-4adb-8fc0-d754b3aa6a63","Type":"ContainerStarted","Data":"7ebb3d9c6d692a802f5352bf935bb3c55698795b85072351a80bf0427dbea6a7"} Sep 16 18:51:20.315193 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:20.315169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" event={"ID":"7fc2e503-0557-4530-b71f-ddeff192e2e3","Type":"ContainerStarted","Data":"2b97fe35c1405aa12f5ee480e415ae644a870c736c89362588e98ddfe5d6811e"} Sep 16 18:51:22.351886 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:22.351827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:23.740627 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:23.740609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 18:51:24.332374 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:24.332344 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" event={"ID":"7fc2e503-0557-4530-b71f-ddeff192e2e3","Type":"ContainerStarted","Data":"a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4"} Sep 16 18:51:24.332572 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:24.332401 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:51:24.333772 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:24.333750 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bcld6" event={"ID":"36905445-43b6-4adb-8fc0-d754b3aa6a63","Type":"ContainerStarted","Data":"3cc7c0a3b48fafbc81126924ec4852c4f9b7a762143c42ed6e4afc79ffb68bc0"} Sep 16 18:51:24.333879 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:24.333873 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:24.358665 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:24.358609 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" podStartSLOduration=1.862787597 podStartE2EDuration="5.358590745s" totalImagesPullingTime="3.495803148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:24.357734387 +0000 UTC m=+422.003829203" watchObservedRunningTime="2026-09-16 18:51:24.358590745 +0000 UTC m=+422.004685554" Sep 16 18:51:24.384213 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:24.384165 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-bcld6" podStartSLOduration=1.730785081 podStartE2EDuration="5.384150274s" totalImagesPullingTime="3.653365193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:24.382928245 +0000 UTC m=+422.029023050" watchObservedRunningTime="2026-09-16 18:51:24.384150274 +0000 UTC m=+422.030245080" Sep 16 18:51:30.340142 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:30.340111 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-bcld6" Sep 16 18:51:55.343302 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:51:55.343275 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:52:03.626949 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:03.626908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:03.785759 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:03.785726 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:05.601718 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.601687 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-lptb5"] Sep 16 18:52:05.602170 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.601871 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" podUID="7fc2e503-0557-4530-b71f-ddeff192e2e3" containerName="manager" containerID="cri-o://a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4" gracePeriod=10 Sep 16 18:52:05.638209 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.638183 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-w98w5"] Sep 16 18:52:05.640181 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.640167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:05.644830 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.644807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkft\" (UniqueName: \"kubernetes.io/projected/f982db09-27cd-4427-bff1-ac58a9a9dea5-kube-api-access-kgkft\") pod \"kserve-controller-manager-f8c46b677-w98w5\" (UID: \"f982db09-27cd-4427-bff1-ac58a9a9dea5\") " pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:05.644891 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.644854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f982db09-27cd-4427-bff1-ac58a9a9dea5-cert\") pod \"kserve-controller-manager-f8c46b677-w98w5\" (UID: \"f982db09-27cd-4427-bff1-ac58a9a9dea5\") " pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:05.655781 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.655753 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-w98w5"] Sep 16 18:52:05.745901 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.745871 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkft\" (UniqueName: \"kubernetes.io/projected/f982db09-27cd-4427-bff1-ac58a9a9dea5-kube-api-access-kgkft\") pod \"kserve-controller-manager-f8c46b677-w98w5\" (UID: \"f982db09-27cd-4427-bff1-ac58a9a9dea5\") " pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:05.745901 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.745909 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f982db09-27cd-4427-bff1-ac58a9a9dea5-cert\") pod \"kserve-controller-manager-f8c46b677-w98w5\" (UID: \"f982db09-27cd-4427-bff1-ac58a9a9dea5\") " pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:05.748212 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.748184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f982db09-27cd-4427-bff1-ac58a9a9dea5-cert\") pod \"kserve-controller-manager-f8c46b677-w98w5\" (UID: \"f982db09-27cd-4427-bff1-ac58a9a9dea5\") " pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:05.757590 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.757563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkft\" (UniqueName: \"kubernetes.io/projected/f982db09-27cd-4427-bff1-ac58a9a9dea5-kube-api-access-kgkft\") pod \"kserve-controller-manager-f8c46b677-w98w5\" (UID: \"f982db09-27cd-4427-bff1-ac58a9a9dea5\") " pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:05.830627 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.830605 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:52:05.846924 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.846905 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d82gc\" (UniqueName: \"kubernetes.io/projected/7fc2e503-0557-4530-b71f-ddeff192e2e3-kube-api-access-d82gc\") pod \"7fc2e503-0557-4530-b71f-ddeff192e2e3\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " Sep 16 18:52:05.847099 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.846965 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc2e503-0557-4530-b71f-ddeff192e2e3-cert\") pod \"7fc2e503-0557-4530-b71f-ddeff192e2e3\" (UID: \"7fc2e503-0557-4530-b71f-ddeff192e2e3\") " Sep 16 18:52:05.849298 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.849272 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc2e503-0557-4530-b71f-ddeff192e2e3-kube-api-access-d82gc" (OuterVolumeSpecName: "kube-api-access-d82gc") pod "7fc2e503-0557-4530-b71f-ddeff192e2e3" (UID: "7fc2e503-0557-4530-b71f-ddeff192e2e3"). InnerVolumeSpecName "kube-api-access-d82gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:52:05.849536 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.849505 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc2e503-0557-4530-b71f-ddeff192e2e3-cert" (OuterVolumeSpecName: "cert") pod "7fc2e503-0557-4530-b71f-ddeff192e2e3" (UID: "7fc2e503-0557-4530-b71f-ddeff192e2e3"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:52:05.947643 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.947609 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d82gc\" (UniqueName: \"kubernetes.io/projected/7fc2e503-0557-4530-b71f-ddeff192e2e3-kube-api-access-d82gc\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:52:05.947643 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.947634 2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7fc2e503-0557-4530-b71f-ddeff192e2e3-cert\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:52:05.966501 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:05.966453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:06.234545 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.234464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-w98w5"] Sep 16 18:52:06.236964 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:52:06.236941 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf982db09_27cd_4427_bff1_ac58a9a9dea5.slice/crio-470d8cbd77833aad940f0c67906516a64c49434fee3131e783c0f861a2fa291e WatchSource:0}: Error finding container 470d8cbd77833aad940f0c67906516a64c49434fee3131e783c0f861a2fa291e: Status 404 returned error can't find the container with id 470d8cbd77833aad940f0c67906516a64c49434fee3131e783c0f861a2fa291e Sep 16 18:52:06.475148 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.475111 2576 generic.go:356] "Generic (PLEG): container finished" podID="7fc2e503-0557-4530-b71f-ddeff192e2e3" containerID="a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4" exitCode=0 Sep 16 18:52:06.475293 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.475203 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" Sep 16 18:52:06.475293 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.475250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" event={"ID":"7fc2e503-0557-4530-b71f-ddeff192e2e3","Type":"ContainerDied","Data":"a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4"} Sep 16 18:52:06.475426 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.475307 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-lptb5" event={"ID":"7fc2e503-0557-4530-b71f-ddeff192e2e3","Type":"ContainerDied","Data":"2b97fe35c1405aa12f5ee480e415ae644a870c736c89362588e98ddfe5d6811e"} Sep 16 18:52:06.475426 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.475338 2576 scope.go:117] "RemoveContainer" containerID="a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4" Sep 16 18:52:06.476616 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.476593 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-w98w5" event={"ID":"f982db09-27cd-4427-bff1-ac58a9a9dea5","Type":"ContainerStarted","Data":"470d8cbd77833aad940f0c67906516a64c49434fee3131e783c0f861a2fa291e"} Sep 16 18:52:06.484249 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.484233 2576 scope.go:117] "RemoveContainer" containerID="a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4" Sep 16 18:52:06.484558 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:52:06.484518 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4\": container with ID starting with a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4 not found: ID does not exist" containerID="a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4" Sep 16 18:52:06.484558 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.484543 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4"} err="failed to get container status \"a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4\": rpc error: code = NotFound desc = could not find container \"a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4\": container with ID starting with a314e29b250a487518eda9da2ec7e28baf9227a10dad791ab0c56e3f0e98cea4 not found: ID does not exist" Sep 16 18:52:06.497166 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.497136 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-lptb5"] Sep 16 18:52:06.499060 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.499042 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-lptb5"] Sep 16 18:52:06.882731 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:06.882705 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc2e503-0557-4530-b71f-ddeff192e2e3" path="/var/lib/kubelet/pods/7fc2e503-0557-4530-b71f-ddeff192e2e3/volumes" Sep 16 18:52:07.481518 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:07.481472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-w98w5" event={"ID":"f982db09-27cd-4427-bff1-ac58a9a9dea5","Type":"ContainerStarted","Data":"96e513fddc3d6f475467ff29f8544bcab544fd64767ee97c6b5b06f3672f1737"} Sep 16 18:52:07.481712 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:07.481569 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:14.551041 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:14.551007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:14.652606 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:14.652566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:20.872713 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:20.872672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:20.972837 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:20.972783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:22.843857 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:22.843825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:22.938071 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:22.938034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:34.033604 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:34.033563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:38.491051 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:38.491023 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8c46b677-w98w5" Sep 16 18:52:38.507760 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:38.507705 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8c46b677-w98w5" podStartSLOduration=32.895064992 podStartE2EDuration="33.507684763s" totalImagesPullingTime="612.619771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:07.518623578 +0000 UTC m=+465.164718383" watchObservedRunningTime="2026-09-16 18:52:38.507684763 +0000 UTC m=+496.153779573" Sep 16 18:52:39.714087 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:39.714047 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:39.818254 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:39.818221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:39.918357 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:39.918326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:40.104093 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:40.104007 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:41.260526 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:41.260477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:41.358635 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:52:41.358607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:26.939656 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:53:26.939565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:27.039012 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:53:27.038976 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:39.467802 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:53:39.467761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:14.254101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:14.254058 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:14.360151 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:14.360117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:22.796119 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:22.796090 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:54:22.802755 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:22.802736 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:54:23.877548 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:23.877502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:23.953129 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:23.953096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:25.553796 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.553765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:25.763696 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.763656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:25.867088 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.866997 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:25.898899 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.898865 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph"] Sep 16 18:54:25.899182 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.899166 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc2e503-0557-4530-b71f-ddeff192e2e3" containerName="manager" Sep 16 18:54:25.899257 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.899185 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc2e503-0557-4530-b71f-ddeff192e2e3" containerName="manager" Sep 16 18:54:25.899257 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.899248 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc2e503-0557-4530-b71f-ddeff192e2e3" containerName="manager" Sep 16 18:54:25.902061 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.902039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:25.904789 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.904766 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:54:25.904789 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.904786 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aa795-kube-rbac-proxy-sar-config\"" Sep 16 18:54:25.904967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.904790 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:54:25.904967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.904787 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-aa795-predictor-serving-cert\"" Sep 16 18:54:25.904967 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.904814 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-hvpnn\"" Sep 16 18:54:25.912237 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.912211 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph"] Sep 16 18:54:25.945400 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.945371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c0d47b-5238-4c06-8bcc-59b1d6374978-proxy-tls\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:25.945573 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.945408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpzwj\" (UniqueName: \"kubernetes.io/projected/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-api-access-qpzwj\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:25.945573 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.945526 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:25.964248 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:25.964226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:26.046137 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.046108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:26.046341 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.046152 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c0d47b-5238-4c06-8bcc-59b1d6374978-proxy-tls\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:26.046341 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.046201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpzwj\" (UniqueName: \"kubernetes.io/projected/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-api-access-qpzwj\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:26.046929 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.046909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:26.048687 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.048669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c0d47b-5238-4c06-8bcc-59b1d6374978-proxy-tls\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:26.056150 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.056123 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpzwj\" (UniqueName: \"kubernetes.io/projected/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-api-access-qpzwj\") pod \"error-404-isvc-aa795-predictor-787c44f496-x4tph\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:26.063478 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.063451 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:26.163161 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.163130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:26.213355 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.213318 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:26.264130 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.263999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:26.363970 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.363928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:26.486027 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.485950 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph"] Sep 16 18:54:26.489125 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:54:26.489087 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c0d47b_5238_4c06_8bcc_59b1d6374978.slice/crio-8efe1512c78018e6e1cf8a7db7d4898c51d5b61d69c400e389d2a93e2cc8cde5 WatchSource:0}: Error finding container 8efe1512c78018e6e1cf8a7db7d4898c51d5b61d69c400e389d2a93e2cc8cde5: Status 404 returned error can't find the container with id 8efe1512c78018e6e1cf8a7db7d4898c51d5b61d69c400e389d2a93e2cc8cde5 Sep 16 18:54:26.949320 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:26.949281 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" event={"ID":"d4c0d47b-5238-4c06-8bcc-59b1d6374978","Type":"ContainerStarted","Data":"8efe1512c78018e6e1cf8a7db7d4898c51d5b61d69c400e389d2a93e2cc8cde5"} Sep 16 18:54:39.004176 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:39.004137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" event={"ID":"d4c0d47b-5238-4c06-8bcc-59b1d6374978","Type":"ContainerStarted","Data":"e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f"} Sep 16 18:54:41.013187 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:41.013151 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" event={"ID":"d4c0d47b-5238-4c06-8bcc-59b1d6374978","Type":"ContainerStarted","Data":"0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5"} Sep 16 18:54:41.013661 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:41.013371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:41.013661 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:41.013520 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:41.014844 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:41.014817 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:54:41.032930 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:41.032887 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podStartSLOduration=2.067539434 podStartE2EDuration="16.032873409s" totalImagesPullingTime="13.965333975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:41.031552007 +0000 UTC m=+618.677646815" watchObservedRunningTime="2026-09-16 18:54:41.032873409 +0000 UTC m=+618.678968217" Sep 16 18:54:42.017166 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:42.017130 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:54:47.022103 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:47.022028 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:54:47.022611 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:47.022583 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:54:57.022680 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:54:57.022639 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:55:07.023402 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:55:07.023362 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:55:07.956761 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:55:07.956720 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:55:17.022725 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:55:17.022688 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:55:27.023684 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:55:27.023644 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:55:55.634449 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:55:55.634417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:55:57.001348 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:55:57.001296 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:05.703453 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:05.703416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:05.847731 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:05.847685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:05.852916 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:05.852874 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph"] Sep 16 18:56:05.853312 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:05.853276 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" containerID="cri-o://e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f" gracePeriod=30 Sep 16 18:56:05.853381 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:05.853328 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kube-rbac-proxy" containerID="cri-o://0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5" gracePeriod=30 Sep 16 18:56:05.944795 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:05.944764 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:06.047342 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.047307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:06.053153 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.053126 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866"] Sep 16 18:56:06.055927 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.055907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.058617 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.058594 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb09c-kube-rbac-proxy-sar-config\"" Sep 16 18:56:06.058725 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.058594 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-eb09c-predictor-serving-cert\"" Sep 16 18:56:06.067276 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.067254 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866"] Sep 16 18:56:06.145988 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.145957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:06.176770 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.176745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/922928ff-8443-4b04-8bfd-174f8c9a58b0-proxy-tls\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.176927 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.176782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfxwq\" (UniqueName: \"kubernetes.io/projected/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-api-access-zfxwq\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.176927 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.176830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.246442 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.246361 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:06.277387 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.277353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/922928ff-8443-4b04-8bfd-174f8c9a58b0-proxy-tls\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.277567 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.277398 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfxwq\" (UniqueName: \"kubernetes.io/projected/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-api-access-zfxwq\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.277567 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.277429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.278102 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.278080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.279879 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.279860 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/922928ff-8443-4b04-8bfd-174f8c9a58b0-proxy-tls\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.286461 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.286439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfxwq\" (UniqueName: \"kubernetes.io/projected/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-api-access-zfxwq\") pod \"error-404-isvc-eb09c-predictor-748fc5bc5b-xs866\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.312851 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.312822 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerID="0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5" exitCode=2 Sep 16 18:56:06.312955 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.312899 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" event={"ID":"d4c0d47b-5238-4c06-8bcc-59b1d6374978","Type":"ContainerDied","Data":"0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5"} Sep 16 18:56:06.366313 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.366288 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:06.637621 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:06.637594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866"] Sep 16 18:56:06.639819 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:56:06.639793 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922928ff_8443_4b04_8bfd_174f8c9a58b0.slice/crio-fee50cc8e307a2184ca3fa16ada1d983a95d117f3698f69ee089a059bc071dbe WatchSource:0}: Error finding container fee50cc8e307a2184ca3fa16ada1d983a95d117f3698f69ee089a059bc071dbe: Status 404 returned error can't find the container with id fee50cc8e307a2184ca3fa16ada1d983a95d117f3698f69ee089a059bc071dbe Sep 16 18:56:07.018214 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:07.018126 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Sep 16 18:56:07.022776 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:07.022755 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:56:07.318513 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:07.318410 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" event={"ID":"922928ff-8443-4b04-8bfd-174f8c9a58b0","Type":"ContainerStarted","Data":"47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b"} Sep 16 18:56:07.318513 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:07.318465 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" event={"ID":"922928ff-8443-4b04-8bfd-174f8c9a58b0","Type":"ContainerStarted","Data":"43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5"} Sep 16 18:56:07.318513 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:07.318510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:07.318724 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:07.318529 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" event={"ID":"922928ff-8443-4b04-8bfd-174f8c9a58b0","Type":"ContainerStarted","Data":"fee50cc8e307a2184ca3fa16ada1d983a95d117f3698f69ee089a059bc071dbe"} Sep 16 18:56:07.335505 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:07.335443 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podStartSLOduration=1.33542616 podStartE2EDuration="1.33542616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:07.334743218 +0000 UTC m=+704.980838049" watchObservedRunningTime="2026-09-16 18:56:07.33542616 +0000 UTC m=+704.981520965" Sep 16 18:56:08.322391 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.322363 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:08.323929 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.323898 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:56:08.896797 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.896773 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:56:08.996101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.996062 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c0d47b-5238-4c06-8bcc-59b1d6374978-proxy-tls\") pod \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " Sep 16 18:56:08.996298 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.996167 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpzwj\" (UniqueName: \"kubernetes.io/projected/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-api-access-qpzwj\") pod \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " Sep 16 18:56:08.996298 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.996186 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-rbac-proxy-sar-config\") pod \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\" (UID: \"d4c0d47b-5238-4c06-8bcc-59b1d6374978\") " Sep 16 18:56:08.996593 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.996573 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4c0d47b-5238-4c06-8bcc-59b1d6374978" (UID: "d4c0d47b-5238-4c06-8bcc-59b1d6374978"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:56:08.998497 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.998461 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-api-access-qpzwj" (OuterVolumeSpecName: "kube-api-access-qpzwj") pod "d4c0d47b-5238-4c06-8bcc-59b1d6374978" (UID: "d4c0d47b-5238-4c06-8bcc-59b1d6374978"). InnerVolumeSpecName "kube-api-access-qpzwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:56:08.998599 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:08.998579 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c0d47b-5238-4c06-8bcc-59b1d6374978-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4c0d47b-5238-4c06-8bcc-59b1d6374978" (UID: "d4c0d47b-5238-4c06-8bcc-59b1d6374978"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:56:09.097105 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.097023 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpzwj\" (UniqueName: \"kubernetes.io/projected/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-api-access-qpzwj\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:56:09.097105 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.097051 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4c0d47b-5238-4c06-8bcc-59b1d6374978-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:56:09.097105 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.097064 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4c0d47b-5238-4c06-8bcc-59b1d6374978-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 18:56:09.326905 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.326870 2576 generic.go:356] "Generic (PLEG): container finished" podID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerID="e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f" exitCode=0 Sep 16 18:56:09.327324 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.326927 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" Sep 16 18:56:09.327324 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.326955 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" event={"ID":"d4c0d47b-5238-4c06-8bcc-59b1d6374978","Type":"ContainerDied","Data":"e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f"} Sep 16 18:56:09.327324 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.327020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph" event={"ID":"d4c0d47b-5238-4c06-8bcc-59b1d6374978","Type":"ContainerDied","Data":"8efe1512c78018e6e1cf8a7db7d4898c51d5b61d69c400e389d2a93e2cc8cde5"} Sep 16 18:56:09.327324 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.327048 2576 scope.go:117] "RemoveContainer" containerID="0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5" Sep 16 18:56:09.327772 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.327740 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:56:09.335805 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.335789 2576 scope.go:117] "RemoveContainer" containerID="e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f" Sep 16 18:56:09.342782 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.342763 2576 scope.go:117] "RemoveContainer" containerID="0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5" Sep 16 18:56:09.343032 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:56:09.343016 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5\": container with ID starting with 0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5 not found: ID does not exist" containerID="0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5" Sep 16 18:56:09.343101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.343038 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5"} err="failed to get container status \"0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5\": rpc error: code = NotFound desc = could not find container \"0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5\": container with ID starting with 0e6936ad5263725243641d214aca621f680c87934245171c01ebe626017c21e5 not found: ID does not exist" Sep 16 18:56:09.343101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.343053 2576 scope.go:117] "RemoveContainer" containerID="e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f" Sep 16 18:56:09.343279 ip-10-0-138-79 kubenswrapper[2576]: E0916 18:56:09.343266 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f\": container with ID starting with e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f not found: ID does not exist" containerID="e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f" Sep 16 18:56:09.343317 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.343284 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f"} err="failed to get container status \"e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f\": rpc error: code = NotFound desc = could not find container \"e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f\": container with ID starting with e8b3af57cd983ad9aad999a74029abf188471ecd29041c4dbb18caa98afee19f not found: ID does not exist" Sep 16 18:56:09.347790 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.347740 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph"] Sep 16 18:56:09.350435 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:09.350415 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-aa795-predictor-787c44f496-x4tph"] Sep 16 18:56:10.886159 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:10.886128 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" path="/var/lib/kubelet/pods/d4c0d47b-5238-4c06-8bcc-59b1d6374978/volumes" Sep 16 18:56:14.331758 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:14.331690 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:14.332186 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:14.332159 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:56:24.333048 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:24.333007 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:56:34.332425 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:34.332385 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:56:35.583319 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:35.583284 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:35.712590 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:35.712557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:44.332099 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:44.332058 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:56:45.672131 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:45.672083 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:45.793152 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:45.793102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:45.882414 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:45.882384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:45.960402 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:45.960306 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:46.036234 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.036201 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:46.039780 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.039755 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44"] Sep 16 18:56:46.040101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.040078 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kube-rbac-proxy" Sep 16 18:56:46.040101 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.040094 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kube-rbac-proxy" Sep 16 18:56:46.040227 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.040108 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" Sep 16 18:56:46.040227 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.040114 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" Sep 16 18:56:46.040227 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.040172 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kube-rbac-proxy" Sep 16 18:56:46.040227 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.040181 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4c0d47b-5238-4c06-8bcc-59b1d6374978" containerName="kserve-container" Sep 16 18:56:46.045004 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.044990 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.047174 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.047153 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2796f-predictor-serving-cert\"" Sep 16 18:56:46.047174 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.047171 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-2796f-kube-rbac-proxy-sar-config\"" Sep 16 18:56:46.053302 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.053288 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44"] Sep 16 18:56:46.148164 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.148127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:46.174260 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.174227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.174397 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.174277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9452c4a7-5a7a-444a-800f-d1d8695c0140-proxy-tls\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.174397 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.174366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjpc\" (UniqueName: \"kubernetes.io/projected/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-api-access-xcjpc\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.275033 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.274934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9452c4a7-5a7a-444a-800f-d1d8695c0140-proxy-tls\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.275033 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.275006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjpc\" (UniqueName: \"kubernetes.io/projected/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-api-access-xcjpc\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.275033 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.275030 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.275801 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.275768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.277534 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.277518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9452c4a7-5a7a-444a-800f-d1d8695c0140-proxy-tls\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.283017 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.282999 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjpc\" (UniqueName: \"kubernetes.io/projected/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-api-access-xcjpc\") pod \"error-404-isvc-2796f-predictor-cc7c89b74-d9k44\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.356855 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.356823 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:46.404458 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.404428 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:46.629572 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:46.629548 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44"] Sep 16 18:56:46.634840 ip-10-0-138-79 kubenswrapper[2576]: W0916 18:56:46.634815 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9452c4a7_5a7a_444a_800f_d1d8695c0140.slice/crio-a98e9ff057171b40dcd7ecc2c94d9ab7133042fe658e998f0db011871f6719fc WatchSource:0}: Error finding container a98e9ff057171b40dcd7ecc2c94d9ab7133042fe658e998f0db011871f6719fc: Status 404 returned error can't find the container with id a98e9ff057171b40dcd7ecc2c94d9ab7133042fe658e998f0db011871f6719fc Sep 16 18:56:47.461517 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:47.461462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" event={"ID":"9452c4a7-5a7a-444a-800f-d1d8695c0140","Type":"ContainerStarted","Data":"869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1"} Sep 16 18:56:47.461934 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:47.461554 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" event={"ID":"9452c4a7-5a7a-444a-800f-d1d8695c0140","Type":"ContainerStarted","Data":"cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799"} Sep 16 18:56:47.461934 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:47.461591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" event={"ID":"9452c4a7-5a7a-444a-800f-d1d8695c0140","Type":"ContainerStarted","Data":"a98e9ff057171b40dcd7ecc2c94d9ab7133042fe658e998f0db011871f6719fc"} Sep 16 18:56:47.461934 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:47.461647 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:47.478524 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:47.478456 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podStartSLOduration=1.478437337 podStartE2EDuration="1.478437337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:47.477604923 +0000 UTC m=+745.123699740" watchObservedRunningTime="2026-09-16 18:56:47.478437337 +0000 UTC m=+745.124532144" Sep 16 18:56:48.465538 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:48.465503 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:48.466733 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:48.466709 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 18:56:49.469001 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:49.468962 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 18:56:54.332631 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:54.332596 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 18:56:54.474129 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:54.474105 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:56:54.474655 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:56:54.474632 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 18:57:04.474754 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:04.474712 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 18:57:05.378775 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:05.378741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:05.480997 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:05.480957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:05.949591 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:05.949553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:06.492943 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:06.492906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:14.475292 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:14.475257 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 18:57:15.210868 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:15.210837 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:24.475116 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:24.475079 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 18:57:26.143966 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:26.143935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:34.475338 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:34.475301 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 18:57:45.905038 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:45.904931 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:46.117670 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:57:46.117625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 18:59:22.817973 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:59:22.817901 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 18:59:22.823107 ip-10-0-138-79 kubenswrapper[2576]: I0916 18:59:22.823078 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:00:01.120420 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:00:01.120382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:01:29.062568 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:01:29.062526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:51.244872 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:02:51.244775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:51.309632 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:02:51.309585 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:04:22.841219 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:04:22.841191 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:04:22.843922 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:04:22.843901 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:05:20.579874 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:20.579776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:20.794843 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:20.794804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:20.946376 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:20.946313 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866"] Sep 16 19:05:20.946623 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:20.946600 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" containerID="cri-o://43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5" gracePeriod=30 Sep 16 19:05:20.946703 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:20.946634 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kube-rbac-proxy" containerID="cri-o://47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b" gracePeriod=30 Sep 16 19:05:20.953232 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:20.953207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:21.023722 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.023694 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms"] Sep 16 19:05:21.026838 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.026817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.030584 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.030553 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c6de0-predictor-serving-cert\"" Sep 16 19:05:21.030682 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.030650 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c6de0-kube-rbac-proxy-sar-config\"" Sep 16 19:05:21.041447 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.041423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms"] Sep 16 19:05:21.085870 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.085840 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:21.182937 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.182905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:21.185997 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.185978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7j8\" (UniqueName: \"kubernetes.io/projected/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-api-access-xl7j8\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.186076 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.186013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.186076 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.186048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.243459 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.243379 2576 generic.go:356] "Generic (PLEG): container finished" podID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerID="47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b" exitCode=2 Sep 16 19:05:21.243459 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.243449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" event={"ID":"922928ff-8443-4b04-8bfd-174f8c9a58b0","Type":"ContainerDied","Data":"47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b"} Sep 16 19:05:21.286802 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.286766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.286967 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.286813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7j8\" (UniqueName: \"kubernetes.io/projected/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-api-access-xl7j8\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.286967 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.286937 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.287078 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:05:21.287064 2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-serving-cert: secret "error-404-isvc-c6de0-predictor-serving-cert" not found Sep 16 19:05:21.287140 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:05:21.287132 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls podName:81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f nodeName:}" failed. No retries permitted until 2026-09-16 19:05:21.787115593 +0000 UTC m=+1259.433210377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls") pod "error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" (UID: "81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f") : secret "error-404-isvc-c6de0-predictor-serving-cert" not found Sep 16 19:05:21.287531 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.287508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.297160 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.297135 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7j8\" (UniqueName: \"kubernetes.io/projected/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-api-access-xl7j8\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.547612 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.547533 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:21.791776 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.791742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.794214 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.794187 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls\") pod \"error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:21.936634 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:21.936602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:22.210060 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:22.209969 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms"] Sep 16 19:05:22.212403 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:05:22.212363 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81e01fdf_b5fe_46ce_b9e4_c515abdfdd2f.slice/crio-3a3ac5ed0be57f1909bba1d677da0b7081f57cab30e239638fe25cd15a6226cd WatchSource:0}: Error finding container 3a3ac5ed0be57f1909bba1d677da0b7081f57cab30e239638fe25cd15a6226cd: Status 404 returned error can't find the container with id 3a3ac5ed0be57f1909bba1d677da0b7081f57cab30e239638fe25cd15a6226cd Sep 16 19:05:22.247720 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:22.247693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" event={"ID":"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f","Type":"ContainerStarted","Data":"3a3ac5ed0be57f1909bba1d677da0b7081f57cab30e239638fe25cd15a6226cd"} Sep 16 19:05:23.254270 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:23.254233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" event={"ID":"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f","Type":"ContainerStarted","Data":"529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527"} Sep 16 19:05:23.254696 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:23.254328 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" event={"ID":"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f","Type":"ContainerStarted","Data":"20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f"} Sep 16 19:05:23.254696 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:23.254419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:23.274289 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:23.274249 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podStartSLOduration=2.274236224 podStartE2EDuration="2.274236224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:23.272656595 +0000 UTC m=+1260.918751403" watchObservedRunningTime="2026-09-16 19:05:23.274236224 +0000 UTC m=+1260.920331029" Sep 16 19:05:24.200067 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.200045 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 19:05:24.259762 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.259731 2576 generic.go:356] "Generic (PLEG): container finished" podID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerID="43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5" exitCode=0 Sep 16 19:05:24.260201 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.259790 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" Sep 16 19:05:24.260201 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.259808 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" event={"ID":"922928ff-8443-4b04-8bfd-174f8c9a58b0","Type":"ContainerDied","Data":"43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5"} Sep 16 19:05:24.260201 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.259867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866" event={"ID":"922928ff-8443-4b04-8bfd-174f8c9a58b0","Type":"ContainerDied","Data":"fee50cc8e307a2184ca3fa16ada1d983a95d117f3698f69ee089a059bc071dbe"} Sep 16 19:05:24.260201 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.259898 2576 scope.go:117] "RemoveContainer" containerID="47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b" Sep 16 19:05:24.260201 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.260148 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:24.261249 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.261227 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:05:24.267764 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.267750 2576 scope.go:117] "RemoveContainer" containerID="43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5" Sep 16 19:05:24.274976 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.274955 2576 scope.go:117] "RemoveContainer" containerID="47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b" Sep 16 19:05:24.275257 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:05:24.275236 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b\": container with ID starting with 47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b not found: ID does not exist" containerID="47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b" Sep 16 19:05:24.275333 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.275270 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b"} err="failed to get container status \"47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b\": rpc error: code = NotFound desc = could not find container \"47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b\": container with ID starting with 47f418871fbeffc150639ac83090130705e98fdefe621d5c255f0bb6d97f4c0b not found: ID does not exist" Sep 16 19:05:24.275333 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.275292 2576 scope.go:117] "RemoveContainer" containerID="43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5" Sep 16 19:05:24.275594 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:05:24.275571 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5\": container with ID starting with 43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5 not found: ID does not exist" containerID="43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5" Sep 16 19:05:24.275647 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.275604 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5"} err="failed to get container status \"43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5\": rpc error: code = NotFound desc = could not find container \"43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5\": container with ID starting with 43afb8600a0ef64f5edfab86a016972cc1a8ab9e0651f57799559f68dd9886a5 not found: ID does not exist" Sep 16 19:05:24.312022 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.311952 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-rbac-proxy-sar-config\") pod \"922928ff-8443-4b04-8bfd-174f8c9a58b0\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " Sep 16 19:05:24.312022 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.311992 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfxwq\" (UniqueName: \"kubernetes.io/projected/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-api-access-zfxwq\") pod \"922928ff-8443-4b04-8bfd-174f8c9a58b0\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " Sep 16 19:05:24.312195 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.312035 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/922928ff-8443-4b04-8bfd-174f8c9a58b0-proxy-tls\") pod \"922928ff-8443-4b04-8bfd-174f8c9a58b0\" (UID: \"922928ff-8443-4b04-8bfd-174f8c9a58b0\") " Sep 16 19:05:24.312345 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.312325 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "922928ff-8443-4b04-8bfd-174f8c9a58b0" (UID: "922928ff-8443-4b04-8bfd-174f8c9a58b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:05:24.314160 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.314133 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-api-access-zfxwq" (OuterVolumeSpecName: "kube-api-access-zfxwq") pod "922928ff-8443-4b04-8bfd-174f8c9a58b0" (UID: "922928ff-8443-4b04-8bfd-174f8c9a58b0"). InnerVolumeSpecName "kube-api-access-zfxwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:05:24.314258 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.314236 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/922928ff-8443-4b04-8bfd-174f8c9a58b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "922928ff-8443-4b04-8bfd-174f8c9a58b0" (UID: "922928ff-8443-4b04-8bfd-174f8c9a58b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:05:24.412936 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.412882 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:05:24.412936 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.412928 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfxwq\" (UniqueName: \"kubernetes.io/projected/922928ff-8443-4b04-8bfd-174f8c9a58b0-kube-api-access-zfxwq\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:05:24.412936 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.412943 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/922928ff-8443-4b04-8bfd-174f8c9a58b0-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:05:24.582966 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.582932 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866"] Sep 16 19:05:24.589944 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.589920 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-eb09c-predictor-748fc5bc5b-xs866"] Sep 16 19:05:24.882102 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:24.882060 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" path="/var/lib/kubelet/pods/922928ff-8443-4b04-8bfd-174f8c9a58b0/volumes" Sep 16 19:05:25.264239 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:25.264154 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:05:30.268652 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:30.268623 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:05:30.269174 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:30.269149 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:05:40.269392 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:40.269352 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:05:50.269922 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:05:50.269884 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:06:00.269750 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.269706 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:06:00.528426 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.528348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:00.668373 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.668337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:00.686195 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.686164 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44"] Sep 16 19:06:00.686541 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.686469 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" containerID="cri-o://cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799" gracePeriod=30 Sep 16 19:06:00.686703 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.686515 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kube-rbac-proxy" containerID="cri-o://869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1" gracePeriod=30 Sep 16 19:06:00.754825 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.754790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:00.971842 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.971810 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:00.990932 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.990906 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v"] Sep 16 19:06:00.991185 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.991175 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kube-rbac-proxy" Sep 16 19:06:00.991231 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.991187 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kube-rbac-proxy" Sep 16 19:06:00.991231 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.991198 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" Sep 16 19:06:00.991231 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.991203 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" Sep 16 19:06:00.991323 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.991241 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kserve-container" Sep 16 19:06:00.991323 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.991248 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="922928ff-8443-4b04-8bfd-174f8c9a58b0" containerName="kube-rbac-proxy" Sep 16 19:06:00.994278 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.994256 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:00.999271 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.999252 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f64b9-kube-rbac-proxy-sar-config\"" Sep 16 19:06:00.999380 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:00.999251 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f64b9-predictor-serving-cert\"" Sep 16 19:06:01.013555 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.013526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v"] Sep 16 19:06:01.083041 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.082968 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm4l8\" (UniqueName: \"kubernetes.io/projected/b057a297-058b-4d65-a071-75f470208355-kube-api-access-xm4l8\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.083041 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.083011 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b057a297-058b-4d65-a071-75f470208355-proxy-tls\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.083213 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.083047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b057a297-058b-4d65-a071-75f470208355-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.171962 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.171926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:01.183623 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.183597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm4l8\" (UniqueName: \"kubernetes.io/projected/b057a297-058b-4d65-a071-75f470208355-kube-api-access-xm4l8\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.183761 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.183637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b057a297-058b-4d65-a071-75f470208355-proxy-tls\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.183761 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.183667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b057a297-058b-4d65-a071-75f470208355-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.184272 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.184254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b057a297-058b-4d65-a071-75f470208355-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.186254 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.186236 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b057a297-058b-4d65-a071-75f470208355-proxy-tls\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.192066 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.192045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm4l8\" (UniqueName: \"kubernetes.io/projected/b057a297-058b-4d65-a071-75f470208355-kube-api-access-xm4l8\") pod \"error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.271144 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.271109 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:01.304398 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.304373 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:01.389905 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.389875 2576 generic.go:356] "Generic (PLEG): container finished" podID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerID="869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1" exitCode=2 Sep 16 19:06:01.390043 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.389933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" event={"ID":"9452c4a7-5a7a-444a-800f-d1d8695c0140","Type":"ContainerDied","Data":"869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1"} Sep 16 19:06:01.579379 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:01.579341 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v"] Sep 16 19:06:01.582183 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:06:01.582154 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb057a297_058b_4d65_a071_75f470208355.slice/crio-25eada73255651c434138629636e811c28e392fb499c429eb53d36acfe0a2b4a WatchSource:0}: Error finding container 25eada73255651c434138629636e811c28e392fb499c429eb53d36acfe0a2b4a: Status 404 returned error can't find the container with id 25eada73255651c434138629636e811c28e392fb499c429eb53d36acfe0a2b4a Sep 16 19:06:02.395289 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:02.395255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" event={"ID":"b057a297-058b-4d65-a071-75f470208355","Type":"ContainerStarted","Data":"9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8"} Sep 16 19:06:02.395794 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:02.395319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" event={"ID":"b057a297-058b-4d65-a071-75f470208355","Type":"ContainerStarted","Data":"4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932"} Sep 16 19:06:02.395794 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:02.395473 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:02.395794 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:02.395533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" event={"ID":"b057a297-058b-4d65-a071-75f470208355","Type":"ContainerStarted","Data":"25eada73255651c434138629636e811c28e392fb499c429eb53d36acfe0a2b4a"} Sep 16 19:06:02.395794 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:02.395560 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:02.396651 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:02.396629 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" Sep 16 19:06:02.412133 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:02.412092 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podStartSLOduration=2.412079031 podStartE2EDuration="2.412079031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:02.411279358 +0000 UTC m=+1300.057374190" watchObservedRunningTime="2026-09-16 19:06:02.412079031 +0000 UTC m=+1300.058173833" Sep 16 19:06:03.399435 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.399395 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" Sep 16 19:06:03.826393 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.826366 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 19:06:03.903143 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.903105 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-rbac-proxy-sar-config\") pod \"9452c4a7-5a7a-444a-800f-d1d8695c0140\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " Sep 16 19:06:03.903143 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.903145 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcjpc\" (UniqueName: \"kubernetes.io/projected/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-api-access-xcjpc\") pod \"9452c4a7-5a7a-444a-800f-d1d8695c0140\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " Sep 16 19:06:03.903398 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.903206 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9452c4a7-5a7a-444a-800f-d1d8695c0140-proxy-tls\") pod \"9452c4a7-5a7a-444a-800f-d1d8695c0140\" (UID: \"9452c4a7-5a7a-444a-800f-d1d8695c0140\") " Sep 16 19:06:03.903524 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.903500 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9452c4a7-5a7a-444a-800f-d1d8695c0140" (UID: "9452c4a7-5a7a-444a-800f-d1d8695c0140"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:06:03.905410 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.905361 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9452c4a7-5a7a-444a-800f-d1d8695c0140-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9452c4a7-5a7a-444a-800f-d1d8695c0140" (UID: "9452c4a7-5a7a-444a-800f-d1d8695c0140"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:06:03.905523 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:03.905499 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-api-access-xcjpc" (OuterVolumeSpecName: "kube-api-access-xcjpc") pod "9452c4a7-5a7a-444a-800f-d1d8695c0140" (UID: "9452c4a7-5a7a-444a-800f-d1d8695c0140"). InnerVolumeSpecName "kube-api-access-xcjpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:06:04.004701 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.004648 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:06:04.004701 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.004689 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjpc\" (UniqueName: \"kubernetes.io/projected/9452c4a7-5a7a-444a-800f-d1d8695c0140-kube-api-access-xcjpc\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:06:04.004701 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.004703 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9452c4a7-5a7a-444a-800f-d1d8695c0140-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:06:04.404428 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.404399 2576 generic.go:356] "Generic (PLEG): container finished" podID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerID="cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799" exitCode=0 Sep 16 19:06:04.404893 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.404436 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" event={"ID":"9452c4a7-5a7a-444a-800f-d1d8695c0140","Type":"ContainerDied","Data":"cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799"} Sep 16 19:06:04.404893 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.404449 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" Sep 16 19:06:04.404893 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.404494 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44" event={"ID":"9452c4a7-5a7a-444a-800f-d1d8695c0140","Type":"ContainerDied","Data":"a98e9ff057171b40dcd7ecc2c94d9ab7133042fe658e998f0db011871f6719fc"} Sep 16 19:06:04.404893 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.404520 2576 scope.go:117] "RemoveContainer" containerID="869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1" Sep 16 19:06:04.412726 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.412699 2576 scope.go:117] "RemoveContainer" containerID="cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799" Sep 16 19:06:04.420019 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.420002 2576 scope.go:117] "RemoveContainer" containerID="869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1" Sep 16 19:06:04.420298 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:06:04.420280 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1\": container with ID starting with 869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1 not found: ID does not exist" containerID="869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1" Sep 16 19:06:04.420345 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.420306 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1"} err="failed to get container status \"869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1\": rpc error: code = NotFound desc = could not find container \"869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1\": container with ID starting with 869c66c36d15edf62a2140c91b7d0b56043a0b9b708c3cc7fc2983a4fc80c3a1 not found: ID does not exist" Sep 16 19:06:04.420345 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.420323 2576 scope.go:117] "RemoveContainer" containerID="cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799" Sep 16 19:06:04.420592 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:06:04.420574 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799\": container with ID starting with cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799 not found: ID does not exist" containerID="cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799" Sep 16 19:06:04.420656 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.420601 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799"} err="failed to get container status \"cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799\": rpc error: code = NotFound desc = could not find container \"cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799\": container with ID starting with cf80fe50840b341fd4ed96ff2fefbf6b4451e55e5cb4ee47a8f7c86e387e3799 not found: ID does not exist" Sep 16 19:06:04.426514 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.426476 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44"] Sep 16 19:06:04.435784 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.435764 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-2796f-predictor-cc7c89b74-d9k44"] Sep 16 19:06:04.883218 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:04.883190 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" path="/var/lib/kubelet/pods/9452c4a7-5a7a-444a-800f-d1d8695c0140/volumes" Sep 16 19:06:08.404272 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:08.404244 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:08.404739 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:08.404661 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" Sep 16 19:06:10.269721 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:10.269695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:06:18.405575 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:18.405532 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" Sep 16 19:06:20.794775 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:20.794739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:21.787650 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:21.787603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:28.404750 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:28.404711 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" Sep 16 19:06:30.906369 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:30.906340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:31.014018 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.013985 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms"] Sep 16 19:06:31.014298 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.014269 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" containerID="cri-o://20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f" gracePeriod=30 Sep 16 19:06:31.014401 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.014308 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kube-rbac-proxy" containerID="cri-o://529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527" gracePeriod=30 Sep 16 19:06:31.018165 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.018135 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:31.182805 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.182731 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn"] Sep 16 19:06:31.183009 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.182997 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" Sep 16 19:06:31.183062 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.183010 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" Sep 16 19:06:31.183062 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.183028 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kube-rbac-proxy" Sep 16 19:06:31.183062 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.183033 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kube-rbac-proxy" Sep 16 19:06:31.183152 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.183077 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kserve-container" Sep 16 19:06:31.183152 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.183084 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9452c4a7-5a7a-444a-800f-d1d8695c0140" containerName="kube-rbac-proxy" Sep 16 19:06:31.185999 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.185983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.189600 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.189562 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3448-predictor-serving-cert\"" Sep 16 19:06:31.189827 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.189813 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3448-kube-rbac-proxy-sar-config\"" Sep 16 19:06:31.200834 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.200807 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn"] Sep 16 19:06:31.306845 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.306803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x25mq\" (UniqueName: \"kubernetes.io/projected/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-api-access-x25mq\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.307024 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.306852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.307024 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.306921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.407965 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.407929 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x25mq\" (UniqueName: \"kubernetes.io/projected/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-api-access-x25mq\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.407965 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.407963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.408209 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.407990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.408209 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:06:31.408105 2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a3448-predictor-serving-cert: secret "error-404-isvc-a3448-predictor-serving-cert" not found Sep 16 19:06:31.408209 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:06:31.408170 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls podName:b3bef913-3e11-4581-a276-0a0bb5be512d nodeName:}" failed. No retries permitted until 2026-09-16 19:06:31.908153251 +0000 UTC m=+1329.554248035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls") pod "error-404-isvc-a3448-predictor-d94964498-cw5sn" (UID: "b3bef913-3e11-4581-a276-0a0bb5be512d") : secret "error-404-isvc-a3448-predictor-serving-cert" not found Sep 16 19:06:31.408647 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.408630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.423256 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.423226 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x25mq\" (UniqueName: \"kubernetes.io/projected/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-api-access-x25mq\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.502388 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.502317 2576 generic.go:356] "Generic (PLEG): container finished" podID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerID="529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527" exitCode=2 Sep 16 19:06:31.502388 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.502365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" event={"ID":"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f","Type":"ContainerDied","Data":"529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527"} Sep 16 19:06:31.912432 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.912400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:31.915033 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:31.915012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls\") pod \"error-404-isvc-a3448-predictor-d94964498-cw5sn\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:32.095334 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.095299 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:32.330848 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.330765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:32.370630 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.370605 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn"] Sep 16 19:06:32.372766 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:06:32.372740 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bef913_3e11_4581_a276_0a0bb5be512d.slice/crio-d0134247c7e36689d60f86c931e1d12e077d560c06edd6902141ac152d32f3b2 WatchSource:0}: Error finding container d0134247c7e36689d60f86c931e1d12e077d560c06edd6902141ac152d32f3b2: Status 404 returned error can't find the container with id d0134247c7e36689d60f86c931e1d12e077d560c06edd6902141ac152d32f3b2 Sep 16 19:06:32.507900 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.507870 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" event={"ID":"b3bef913-3e11-4581-a276-0a0bb5be512d","Type":"ContainerStarted","Data":"f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15"} Sep 16 19:06:32.508037 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.507922 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" event={"ID":"b3bef913-3e11-4581-a276-0a0bb5be512d","Type":"ContainerStarted","Data":"c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f"} Sep 16 19:06:32.508037 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.507942 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" event={"ID":"b3bef913-3e11-4581-a276-0a0bb5be512d","Type":"ContainerStarted","Data":"d0134247c7e36689d60f86c931e1d12e077d560c06edd6902141ac152d32f3b2"} Sep 16 19:06:32.508037 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.507966 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:32.508037 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.507993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:32.509256 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.509235 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:06:32.528214 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:32.528175 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podStartSLOduration=1.5281605900000002 podStartE2EDuration="1.52816059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:32.527764619 +0000 UTC m=+1330.173859433" watchObservedRunningTime="2026-09-16 19:06:32.52816059 +0000 UTC m=+1330.174255395" Sep 16 19:06:33.511531 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:33.511471 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:06:34.264771 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.264748 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:06:34.327410 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.327337 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl7j8\" (UniqueName: \"kubernetes.io/projected/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-api-access-xl7j8\") pod \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " Sep 16 19:06:34.327410 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.327403 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-rbac-proxy-sar-config\") pod \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " Sep 16 19:06:34.327633 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.327454 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls\") pod \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\" (UID: \"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f\") " Sep 16 19:06:34.327825 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.327802 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" (UID: "81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:06:34.329601 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.329575 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-api-access-xl7j8" (OuterVolumeSpecName: "kube-api-access-xl7j8") pod "81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" (UID: "81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f"). InnerVolumeSpecName "kube-api-access-xl7j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:06:34.329601 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.329590 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" (UID: "81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:06:34.428832 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.428802 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:06:34.428832 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.428829 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl7j8\" (UniqueName: \"kubernetes.io/projected/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-api-access-xl7j8\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:06:34.428832 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.428839 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:06:34.516782 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.516747 2576 generic.go:356] "Generic (PLEG): container finished" podID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerID="20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f" exitCode=0 Sep 16 19:06:34.517153 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.516796 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" Sep 16 19:06:34.517153 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.516828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" event={"ID":"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f","Type":"ContainerDied","Data":"20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f"} Sep 16 19:06:34.517153 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.516889 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms" event={"ID":"81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f","Type":"ContainerDied","Data":"3a3ac5ed0be57f1909bba1d677da0b7081f57cab30e239638fe25cd15a6226cd"} Sep 16 19:06:34.517153 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.516915 2576 scope.go:117] "RemoveContainer" containerID="529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527" Sep 16 19:06:34.525727 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.525697 2576 scope.go:117] "RemoveContainer" containerID="20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f" Sep 16 19:06:34.536578 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.536556 2576 scope.go:117] "RemoveContainer" containerID="529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527" Sep 16 19:06:34.536893 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:06:34.536875 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527\": container with ID starting with 529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527 not found: ID does not exist" containerID="529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527" Sep 16 19:06:34.536944 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.536902 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527"} err="failed to get container status \"529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527\": rpc error: code = NotFound desc = could not find container \"529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527\": container with ID starting with 529a0283742a20a49cf9b42927306edae64bcde609d592a414d629834ad41527 not found: ID does not exist" Sep 16 19:06:34.536944 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.536920 2576 scope.go:117] "RemoveContainer" containerID="20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f" Sep 16 19:06:34.537165 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:06:34.537147 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f\": container with ID starting with 20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f not found: ID does not exist" containerID="20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f" Sep 16 19:06:34.537204 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.537172 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f"} err="failed to get container status \"20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f\": rpc error: code = NotFound desc = could not find container \"20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f\": container with ID starting with 20673243b1a1d6ed95a9577b7d3419f8dae9eda79d3e778ecc1452111816e80f not found: ID does not exist" Sep 16 19:06:34.544391 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.544370 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms"] Sep 16 19:06:34.549284 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.549256 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6de0-predictor-5ff88b4bf4-kdvms"] Sep 16 19:06:34.882591 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:34.882559 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" path="/var/lib/kubelet/pods/81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f/volumes" Sep 16 19:06:38.404936 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:38.404896 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused" Sep 16 19:06:38.516500 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:38.516455 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:06:38.516883 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:38.516859 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:06:48.406136 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:48.406057 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:06:48.517325 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:48.517291 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:06:53.373406 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:53.373366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:06:58.517074 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:06:58.517035 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:07:00.753528 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:00.753470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:01.321558 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:01.321528 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:08.517418 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:08.517382 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:07:10.861074 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:10.861039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:10.965583 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:10.965547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:10.981921 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:10.981890 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v"] Sep 16 19:07:10.982173 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:10.982148 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" containerID="cri-o://4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932" gracePeriod=30 Sep 16 19:07:10.982248 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:10.982149 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kube-rbac-proxy" containerID="cri-o://9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8" gracePeriod=30 Sep 16 19:07:11.047674 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.047643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:11.128836 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.128803 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx"] Sep 16 19:07:11.129102 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.129090 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kube-rbac-proxy" Sep 16 19:07:11.129143 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.129103 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kube-rbac-proxy" Sep 16 19:07:11.129143 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.129117 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" Sep 16 19:07:11.129143 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.129122 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" Sep 16 19:07:11.129237 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.129170 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kserve-container" Sep 16 19:07:11.129237 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.129177 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e01fdf-b5fe-46ce-b9e4-c515abdfdd2f" containerName="kube-rbac-proxy" Sep 16 19:07:11.131606 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.131586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:11.132073 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.132060 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.134253 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.134235 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e99c4-predictor-serving-cert\"" Sep 16 19:07:11.134366 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.134268 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e99c4-kube-rbac-proxy-sar-config\"" Sep 16 19:07:11.145127 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.145105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx"] Sep 16 19:07:11.215238 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.215195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fnr\" (UniqueName: \"kubernetes.io/projected/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-api-access-l9fnr\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.215409 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.215261 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd4c11-16ff-431d-8dbb-6d01582305b8-proxy-tls\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.215409 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.215298 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.245474 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.245444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:11.316338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.316312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd4c11-16ff-431d-8dbb-6d01582305b8-proxy-tls\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.316504 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.316356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.316504 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.316401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fnr\" (UniqueName: \"kubernetes.io/projected/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-api-access-l9fnr\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.317033 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.317006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.318908 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.318887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd4c11-16ff-431d-8dbb-6d01582305b8-proxy-tls\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.325159 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.325140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fnr\" (UniqueName: \"kubernetes.io/projected/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-api-access-l9fnr\") pod \"error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.390950 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.390847 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:11.442496 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.442458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:11.659132 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.659056 2576 generic.go:356] "Generic (PLEG): container finished" podID="b057a297-058b-4d65-a071-75f470208355" containerID="9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8" exitCode=2 Sep 16 19:07:11.659132 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.659127 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" event={"ID":"b057a297-058b-4d65-a071-75f470208355","Type":"ContainerDied","Data":"9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8"} Sep 16 19:07:11.713241 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:11.713209 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx"] Sep 16 19:07:11.716001 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:07:11.715972 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fd4c11_16ff_431d_8dbb_6d01582305b8.slice/crio-b700119503209716c7275cfd635e3aa005312b078befe7ef87b40028f489c9f2 WatchSource:0}: Error finding container b700119503209716c7275cfd635e3aa005312b078befe7ef87b40028f489c9f2: Status 404 returned error can't find the container with id b700119503209716c7275cfd635e3aa005312b078befe7ef87b40028f489c9f2 Sep 16 19:07:12.664998 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:12.664970 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" event={"ID":"30fd4c11-16ff-431d-8dbb-6d01582305b8","Type":"ContainerStarted","Data":"1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c"} Sep 16 19:07:12.665368 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:12.665021 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" event={"ID":"30fd4c11-16ff-431d-8dbb-6d01582305b8","Type":"ContainerStarted","Data":"ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df"} Sep 16 19:07:12.665368 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:12.665041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" event={"ID":"30fd4c11-16ff-431d-8dbb-6d01582305b8","Type":"ContainerStarted","Data":"b700119503209716c7275cfd635e3aa005312b078befe7ef87b40028f489c9f2"} Sep 16 19:07:12.665368 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:12.665235 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:12.665368 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:12.665360 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:12.666718 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:12.666694 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:07:12.703288 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:12.703242 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podStartSLOduration=1.7032297650000001 podStartE2EDuration="1.703229765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:12.700785941 +0000 UTC m=+1370.346880747" watchObservedRunningTime="2026-09-16 19:07:12.703229765 +0000 UTC m=+1370.349324615" Sep 16 19:07:13.399845 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:13.399797 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused" Sep 16 19:07:13.668604 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:13.668516 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:07:14.121961 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.121939 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:07:14.138205 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.138188 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b057a297-058b-4d65-a071-75f470208355-kube-rbac-proxy-sar-config\") pod \"b057a297-058b-4d65-a071-75f470208355\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " Sep 16 19:07:14.138302 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.138254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b057a297-058b-4d65-a071-75f470208355-proxy-tls\") pod \"b057a297-058b-4d65-a071-75f470208355\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " Sep 16 19:07:14.138364 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.138299 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm4l8\" (UniqueName: \"kubernetes.io/projected/b057a297-058b-4d65-a071-75f470208355-kube-api-access-xm4l8\") pod \"b057a297-058b-4d65-a071-75f470208355\" (UID: \"b057a297-058b-4d65-a071-75f470208355\") " Sep 16 19:07:14.138582 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.138553 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b057a297-058b-4d65-a071-75f470208355-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b057a297-058b-4d65-a071-75f470208355" (UID: "b057a297-058b-4d65-a071-75f470208355"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:14.140416 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.140393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b057a297-058b-4d65-a071-75f470208355-kube-api-access-xm4l8" (OuterVolumeSpecName: "kube-api-access-xm4l8") pod "b057a297-058b-4d65-a071-75f470208355" (UID: "b057a297-058b-4d65-a071-75f470208355"). InnerVolumeSpecName "kube-api-access-xm4l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:07:14.140525 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.140409 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b057a297-058b-4d65-a071-75f470208355-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b057a297-058b-4d65-a071-75f470208355" (UID: "b057a297-058b-4d65-a071-75f470208355"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:14.239007 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.238923 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b057a297-058b-4d65-a071-75f470208355-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:07:14.239007 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.238955 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm4l8\" (UniqueName: \"kubernetes.io/projected/b057a297-058b-4d65-a071-75f470208355-kube-api-access-xm4l8\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:07:14.239007 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.238966 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b057a297-058b-4d65-a071-75f470208355-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:07:14.673214 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.673173 2576 generic.go:356] "Generic (PLEG): container finished" podID="b057a297-058b-4d65-a071-75f470208355" containerID="4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932" exitCode=0 Sep 16 19:07:14.673711 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.673231 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" Sep 16 19:07:14.673711 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.673265 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" event={"ID":"b057a297-058b-4d65-a071-75f470208355","Type":"ContainerDied","Data":"4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932"} Sep 16 19:07:14.673711 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.673349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v" event={"ID":"b057a297-058b-4d65-a071-75f470208355","Type":"ContainerDied","Data":"25eada73255651c434138629636e811c28e392fb499c429eb53d36acfe0a2b4a"} Sep 16 19:07:14.673711 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.673377 2576 scope.go:117] "RemoveContainer" containerID="9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8" Sep 16 19:07:14.683727 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.683699 2576 scope.go:117] "RemoveContainer" containerID="4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932" Sep 16 19:07:14.690934 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.690917 2576 scope.go:117] "RemoveContainer" containerID="9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8" Sep 16 19:07:14.691206 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:07:14.691188 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8\": container with ID starting with 9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8 not found: ID does not exist" containerID="9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8" Sep 16 19:07:14.691267 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.691214 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8"} err="failed to get container status \"9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8\": rpc error: code = NotFound desc = could not find container \"9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8\": container with ID starting with 9060b96770d99e8b1378a0561dd820a4763e16750c292fa44da4b425a5557bc8 not found: ID does not exist" Sep 16 19:07:14.691267 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.691231 2576 scope.go:117] "RemoveContainer" containerID="4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932" Sep 16 19:07:14.691497 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:07:14.691462 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932\": container with ID starting with 4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932 not found: ID does not exist" containerID="4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932" Sep 16 19:07:14.691543 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.691510 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932"} err="failed to get container status \"4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932\": rpc error: code = NotFound desc = could not find container \"4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932\": container with ID starting with 4307bde08b1762c1bc7117df71fc4974c12ea849abbfe673ae0cdb131bd94932 not found: ID does not exist" Sep 16 19:07:14.697076 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.697057 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v"] Sep 16 19:07:14.703094 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.703075 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f64b9-predictor-56fdcf59dd-6cm9v"] Sep 16 19:07:14.882775 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:14.882745 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b057a297-058b-4d65-a071-75f470208355" path="/var/lib/kubelet/pods/b057a297-058b-4d65-a071-75f470208355/volumes" Sep 16 19:07:18.517622 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:18.517593 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:07:18.673004 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:18.672976 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:07:18.673450 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:18.673427 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:07:28.674258 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:28.674221 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:07:31.216220 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:31.216189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:31.365098 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:31.365043 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:37.450163 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:37.450124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:38.673597 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:38.673561 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:07:48.673766 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:48.673729 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:07:58.674258 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:07:58.674225 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:08:11.193851 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:08:11.193815 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:11.332781 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:08:11.332728 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:45.572064 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:08:45.571995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:45.650867 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:08:45.650835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:11.488591 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:09:11.488543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:22.863807 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:09:22.863780 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:09:22.864860 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:09:22.864839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:12:11.770081 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:12:11.770045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:22.064925 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:12:22.064883 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:31.534281 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:12:31.534234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:22.891176 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:14:22.891150 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:14:22.891626 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:14:22.891461 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:15:45.883066 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:45.882994 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:46.001161 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.001124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:46.119678 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.119644 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn"] Sep 16 19:15:46.119893 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.119851 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" containerID="cri-o://c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f" gracePeriod=30 Sep 16 19:15:46.120042 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.119887 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kube-rbac-proxy" containerID="cri-o://f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15" gracePeriod=30 Sep 16 19:15:46.126095 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.126069 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:46.220325 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220232 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4"] Sep 16 19:15:46.220500 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:46.220639 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220625 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" Sep 16 19:15:46.220683 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220642 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" Sep 16 19:15:46.220683 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220656 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kube-rbac-proxy" Sep 16 19:15:46.220683 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220670 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kube-rbac-proxy" Sep 16 19:15:46.220775 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220734 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kserve-container" Sep 16 19:15:46.220775 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.220742 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b057a297-058b-4d65-a071-75f470208355" containerName="kube-rbac-proxy" Sep 16 19:15:46.223667 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.223648 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.226271 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.226250 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-da3eb-predictor-serving-cert\"" Sep 16 19:15:46.226271 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.226250 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-da3eb-kube-rbac-proxy-sar-config\"" Sep 16 19:15:46.237239 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.237218 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4"] Sep 16 19:15:46.330154 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.330114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:46.365722 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.365686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.365873 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.365728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf2wt\" (UniqueName: \"kubernetes.io/projected/1a1c0844-99c2-4196-9069-33194ea2652e-kube-api-access-xf2wt\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.365873 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.365809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0844-99c2-4196-9069-33194ea2652e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.466606 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.466576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.466769 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.466612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf2wt\" (UniqueName: \"kubernetes.io/projected/1a1c0844-99c2-4196-9069-33194ea2652e-kube-api-access-xf2wt\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.466769 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.466641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0844-99c2-4196-9069-33194ea2652e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.466769 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:15:46.466758 2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-serving-cert: secret "error-404-isvc-da3eb-predictor-serving-cert" not found Sep 16 19:15:46.466944 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:15:46.466836 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls podName:1a1c0844-99c2-4196-9069-33194ea2652e nodeName:}" failed. No retries permitted until 2026-09-16 19:15:46.966820059 +0000 UTC m=+1884.612914843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls") pod "error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" (UID: "1a1c0844-99c2-4196-9069-33194ea2652e") : secret "error-404-isvc-da3eb-predictor-serving-cert" not found Sep 16 19:15:46.467361 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.467331 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0844-99c2-4196-9069-33194ea2652e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.471027 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.470969 2576 generic.go:356] "Generic (PLEG): container finished" podID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerID="f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15" exitCode=2 Sep 16 19:15:46.471115 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.471045 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" event={"ID":"b3bef913-3e11-4581-a276-0a0bb5be512d","Type":"ContainerDied","Data":"f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15"} Sep 16 19:15:46.492712 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.492689 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf2wt\" (UniqueName: \"kubernetes.io/projected/1a1c0844-99c2-4196-9069-33194ea2652e-kube-api-access-xf2wt\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.781252 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.781171 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:15:46.970897 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.970858 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:46.973424 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:46.973406 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls\") pod \"error-404-isvc-da3eb-predictor-5c7778cd96-zctq4\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:47.133323 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:47.133289 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:47.418743 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:47.418671 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4"] Sep 16 19:15:47.422230 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:15:47.422198 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1c0844_99c2_4196_9069_33194ea2652e.slice/crio-dcb921818dda4278a2a58d3026c57e62797d9369b00cc155148588dfb53ea0b1 WatchSource:0}: Error finding container dcb921818dda4278a2a58d3026c57e62797d9369b00cc155148588dfb53ea0b1: Status 404 returned error can't find the container with id dcb921818dda4278a2a58d3026c57e62797d9369b00cc155148588dfb53ea0b1 Sep 16 19:15:47.476825 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:47.476800 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" event={"ID":"1a1c0844-99c2-4196-9069-33194ea2652e","Type":"ContainerStarted","Data":"dcb921818dda4278a2a58d3026c57e62797d9369b00cc155148588dfb53ea0b1"} Sep 16 19:15:48.482385 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:48.482351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" event={"ID":"1a1c0844-99c2-4196-9069-33194ea2652e","Type":"ContainerStarted","Data":"b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977"} Sep 16 19:15:48.482784 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:48.482404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" event={"ID":"1a1c0844-99c2-4196-9069-33194ea2652e","Type":"ContainerStarted","Data":"471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3"} Sep 16 19:15:48.482784 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:48.482429 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:48.512468 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:48.512433 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused" Sep 16 19:15:48.516838 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:48.516817 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:15:49.485817 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:49.485782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:49.486969 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:49.486945 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:15:50.489002 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:50.488963 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:15:52.862298 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:52.862275 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:15:52.888345 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:52.888292 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podStartSLOduration=6.888272123 podStartE2EDuration="6.888272123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:48.522636793 +0000 UTC m=+1886.168731609" watchObservedRunningTime="2026-09-16 19:15:52.888272123 +0000 UTC m=+1890.534366929" Sep 16 19:15:53.011265 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.011178 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-rbac-proxy-sar-config\") pod \"b3bef913-3e11-4581-a276-0a0bb5be512d\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " Sep 16 19:15:53.011265 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.011218 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls\") pod \"b3bef913-3e11-4581-a276-0a0bb5be512d\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " Sep 16 19:15:53.011265 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.011242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x25mq\" (UniqueName: \"kubernetes.io/projected/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-api-access-x25mq\") pod \"b3bef913-3e11-4581-a276-0a0bb5be512d\" (UID: \"b3bef913-3e11-4581-a276-0a0bb5be512d\") " Sep 16 19:15:53.011671 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.011641 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3bef913-3e11-4581-a276-0a0bb5be512d" (UID: "b3bef913-3e11-4581-a276-0a0bb5be512d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:15:53.013559 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.013535 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-api-access-x25mq" (OuterVolumeSpecName: "kube-api-access-x25mq") pod "b3bef913-3e11-4581-a276-0a0bb5be512d" (UID: "b3bef913-3e11-4581-a276-0a0bb5be512d"). InnerVolumeSpecName "kube-api-access-x25mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:15:53.013559 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.013548 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3bef913-3e11-4581-a276-0a0bb5be512d" (UID: "b3bef913-3e11-4581-a276-0a0bb5be512d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:15:53.112022 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.111991 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:15:53.112022 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.112014 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3bef913-3e11-4581-a276-0a0bb5be512d-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:15:53.112022 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.112025 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x25mq\" (UniqueName: \"kubernetes.io/projected/b3bef913-3e11-4581-a276-0a0bb5be512d-kube-api-access-x25mq\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:15:53.500045 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.500009 2576 generic.go:356] "Generic (PLEG): container finished" podID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerID="c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f" exitCode=0 Sep 16 19:15:53.500222 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.500066 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" Sep 16 19:15:53.500222 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.500096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" event={"ID":"b3bef913-3e11-4581-a276-0a0bb5be512d","Type":"ContainerDied","Data":"c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f"} Sep 16 19:15:53.500222 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.500166 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn" event={"ID":"b3bef913-3e11-4581-a276-0a0bb5be512d","Type":"ContainerDied","Data":"d0134247c7e36689d60f86c931e1d12e077d560c06edd6902141ac152d32f3b2"} Sep 16 19:15:53.500222 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.500205 2576 scope.go:117] "RemoveContainer" containerID="f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15" Sep 16 19:15:53.508630 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.508615 2576 scope.go:117] "RemoveContainer" containerID="c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f" Sep 16 19:15:53.515665 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.515647 2576 scope.go:117] "RemoveContainer" containerID="f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15" Sep 16 19:15:53.515923 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:15:53.515905 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15\": container with ID starting with f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15 not found: ID does not exist" containerID="f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15" Sep 16 19:15:53.515974 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.515932 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15"} err="failed to get container status \"f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15\": rpc error: code = NotFound desc = could not find container \"f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15\": container with ID starting with f11def46712e16a477e1e5dbbea92b8bc295d15e6855bf51aafb18194d4f2e15 not found: ID does not exist" Sep 16 19:15:53.515974 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.515948 2576 scope.go:117] "RemoveContainer" containerID="c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f" Sep 16 19:15:53.516223 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:15:53.516204 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f\": container with ID starting with c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f not found: ID does not exist" containerID="c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f" Sep 16 19:15:53.516279 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.516229 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f"} err="failed to get container status \"c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f\": rpc error: code = NotFound desc = could not find container \"c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f\": container with ID starting with c04a9537c47a4691fc6c5b5f8df65fe7e1f2d8b01c08fe85fcca0c7cfbcf894f not found: ID does not exist" Sep 16 19:15:53.522324 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.522304 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn"] Sep 16 19:15:53.526505 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:53.526465 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3448-predictor-d94964498-cw5sn"] Sep 16 19:15:54.882364 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:54.882335 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" path="/var/lib/kubelet/pods/b3bef913-3e11-4581-a276-0a0bb5be512d/volumes" Sep 16 19:15:55.494247 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:55.494222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:15:55.494957 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:15:55.494927 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:16:05.495051 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:05.495012 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:16:15.495406 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:15.495368 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:16:25.495519 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.495464 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:16:25.665259 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.665221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:25.749809 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.749723 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx"] Sep 16 19:16:25.750025 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.749997 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" containerID="cri-o://ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df" gracePeriod=30 Sep 16 19:16:25.750131 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.750069 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kube-rbac-proxy" containerID="cri-o://1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c" gracePeriod=30 Sep 16 19:16:25.780723 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.780671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:25.879877 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.879827 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:25.990726 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:25.990692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:26.045515 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.045421 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn"] Sep 16 19:16:26.045729 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.045717 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" Sep 16 19:16:26.045776 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.045731 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" Sep 16 19:16:26.045776 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.045747 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kube-rbac-proxy" Sep 16 19:16:26.045776 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.045752 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kube-rbac-proxy" Sep 16 19:16:26.045868 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.045797 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kserve-container" Sep 16 19:16:26.045868 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.045808 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3bef913-3e11-4581-a276-0a0bb5be512d" containerName="kube-rbac-proxy" Sep 16 19:16:26.055733 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.055713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.058812 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.058791 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ca22e-kube-rbac-proxy-sar-config\"" Sep 16 19:16:26.058989 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.058965 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-ca22e-predictor-serving-cert\"" Sep 16 19:16:26.064829 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.064805 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn"] Sep 16 19:16:26.110020 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.109985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:26.151106 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.151064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szh46\" (UniqueName: \"kubernetes.io/projected/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-api-access-szh46\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.151263 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.151113 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.151263 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.151136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.187027 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.187001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:26.252289 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.252263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szh46\" (UniqueName: \"kubernetes.io/projected/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-api-access-szh46\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.252461 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.252312 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.252461 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.252353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.252603 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:16:26.252504 2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-serving-cert: secret "error-404-isvc-ca22e-predictor-serving-cert" not found Sep 16 19:16:26.252603 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:16:26.252564 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls podName:655b1db8-2a90-41d2-9723-a2d8ed979c8c nodeName:}" failed. No retries permitted until 2026-09-16 19:16:26.752544965 +0000 UTC m=+1924.398639764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls") pod "error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" (UID: "655b1db8-2a90-41d2-9723-a2d8ed979c8c") : secret "error-404-isvc-ca22e-predictor-serving-cert" not found Sep 16 19:16:26.252964 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.252945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.261310 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.261292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szh46\" (UniqueName: \"kubernetes.io/projected/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-api-access-szh46\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.324341 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.324262 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:26.624942 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.624908 2576 generic.go:356] "Generic (PLEG): container finished" podID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerID="1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c" exitCode=2 Sep 16 19:16:26.625279 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.624980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" event={"ID":"30fd4c11-16ff-431d-8dbb-6d01582305b8","Type":"ContainerDied","Data":"1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c"} Sep 16 19:16:26.756650 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.756615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.759199 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.759177 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls\") pod \"error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:26.966021 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:26.965940 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:27.241663 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:27.241636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn"] Sep 16 19:16:27.243170 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:16:27.243145 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655b1db8_2a90_41d2_9723_a2d8ed979c8c.slice/crio-46e04f0afc900c1dcb082ec2919e3a7519b1b47abbf6ad26a0ef6fc8601cb798 WatchSource:0}: Error finding container 46e04f0afc900c1dcb082ec2919e3a7519b1b47abbf6ad26a0ef6fc8601cb798: Status 404 returned error can't find the container with id 46e04f0afc900c1dcb082ec2919e3a7519b1b47abbf6ad26a0ef6fc8601cb798 Sep 16 19:16:27.631272 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:27.631238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" event={"ID":"655b1db8-2a90-41d2-9723-a2d8ed979c8c","Type":"ContainerStarted","Data":"0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d"} Sep 16 19:16:27.631717 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:27.631305 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" event={"ID":"655b1db8-2a90-41d2-9723-a2d8ed979c8c","Type":"ContainerStarted","Data":"a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203"} Sep 16 19:16:27.631717 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:27.631336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" event={"ID":"655b1db8-2a90-41d2-9723-a2d8ed979c8c","Type":"ContainerStarted","Data":"46e04f0afc900c1dcb082ec2919e3a7519b1b47abbf6ad26a0ef6fc8601cb798"} Sep 16 19:16:27.631717 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:27.631511 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:27.649353 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:27.649299 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podStartSLOduration=1.649280422 podStartE2EDuration="1.649280422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:27.647817901 +0000 UTC m=+1925.293912708" watchObservedRunningTime="2026-09-16 19:16:27.649280422 +0000 UTC m=+1925.295375230" Sep 16 19:16:28.634660 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:28.634627 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:28.635913 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:28.635888 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:16:28.668955 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:28.668912 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused" Sep 16 19:16:28.674330 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:28.674308 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:16:29.299783 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.299760 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:16:29.378320 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.378278 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9fnr\" (UniqueName: \"kubernetes.io/projected/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-api-access-l9fnr\") pod \"30fd4c11-16ff-431d-8dbb-6d01582305b8\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " Sep 16 19:16:29.378541 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.378358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd4c11-16ff-431d-8dbb-6d01582305b8-proxy-tls\") pod \"30fd4c11-16ff-431d-8dbb-6d01582305b8\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " Sep 16 19:16:29.378541 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.378418 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-rbac-proxy-sar-config\") pod \"30fd4c11-16ff-431d-8dbb-6d01582305b8\" (UID: \"30fd4c11-16ff-431d-8dbb-6d01582305b8\") " Sep 16 19:16:29.378826 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.378796 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "30fd4c11-16ff-431d-8dbb-6d01582305b8" (UID: "30fd4c11-16ff-431d-8dbb-6d01582305b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:16:29.380570 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.380546 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30fd4c11-16ff-431d-8dbb-6d01582305b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "30fd4c11-16ff-431d-8dbb-6d01582305b8" (UID: "30fd4c11-16ff-431d-8dbb-6d01582305b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:16:29.380681 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.380659 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-api-access-l9fnr" (OuterVolumeSpecName: "kube-api-access-l9fnr") pod "30fd4c11-16ff-431d-8dbb-6d01582305b8" (UID: "30fd4c11-16ff-431d-8dbb-6d01582305b8"). InnerVolumeSpecName "kube-api-access-l9fnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:16:29.479213 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.479113 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/30fd4c11-16ff-431d-8dbb-6d01582305b8-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:16:29.479213 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.479159 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:16:29.479213 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.479170 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fnr\" (UniqueName: \"kubernetes.io/projected/30fd4c11-16ff-431d-8dbb-6d01582305b8-kube-api-access-l9fnr\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:16:29.640380 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.640340 2576 generic.go:356] "Generic (PLEG): container finished" podID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerID="ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df" exitCode=0 Sep 16 19:16:29.640734 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.640405 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" Sep 16 19:16:29.640734 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.640423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" event={"ID":"30fd4c11-16ff-431d-8dbb-6d01582305b8","Type":"ContainerDied","Data":"ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df"} Sep 16 19:16:29.640734 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.640532 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx" event={"ID":"30fd4c11-16ff-431d-8dbb-6d01582305b8","Type":"ContainerDied","Data":"b700119503209716c7275cfd635e3aa005312b078befe7ef87b40028f489c9f2"} Sep 16 19:16:29.640734 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.640561 2576 scope.go:117] "RemoveContainer" containerID="1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c" Sep 16 19:16:29.641258 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.641227 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:16:29.649742 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.649717 2576 scope.go:117] "RemoveContainer" containerID="ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df" Sep 16 19:16:29.657162 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.657147 2576 scope.go:117] "RemoveContainer" containerID="1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c" Sep 16 19:16:29.657458 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:16:29.657440 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c\": container with ID starting with 1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c not found: ID does not exist" containerID="1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c" Sep 16 19:16:29.657513 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.657466 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c"} err="failed to get container status \"1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c\": rpc error: code = NotFound desc = could not find container \"1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c\": container with ID starting with 1f6ea09c0849b5771a09ee3bcee4a2182bfff515323cd573f6077455c9f87d1c not found: ID does not exist" Sep 16 19:16:29.657513 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.657505 2576 scope.go:117] "RemoveContainer" containerID="ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df" Sep 16 19:16:29.657750 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:16:29.657732 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df\": container with ID starting with ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df not found: ID does not exist" containerID="ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df" Sep 16 19:16:29.657784 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.657757 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df"} err="failed to get container status \"ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df\": rpc error: code = NotFound desc = could not find container \"ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df\": container with ID starting with ffe9bc009ef4d0053714a4564f4149cd8fb9faf72af33170b35ca418037ec8df not found: ID does not exist" Sep 16 19:16:29.662460 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.662439 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx"] Sep 16 19:16:29.668088 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:29.668069 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e99c4-predictor-57c6bffb46-jrjhx"] Sep 16 19:16:30.882399 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:30.882371 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" path="/var/lib/kubelet/pods/30fd4c11-16ff-431d-8dbb-6d01582305b8/volumes" Sep 16 19:16:34.645034 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:34.645007 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:16:34.645538 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:34.645416 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:16:35.496225 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:35.496200 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:16:44.645918 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:44.645874 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:16:46.075632 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:46.075598 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:46.237417 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:46.237380 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:54.646187 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:54.646143 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:16:56.140427 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.140352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:56.264909 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.264875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:56.278846 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.278816 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4"] Sep 16 19:16:56.279072 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.279041 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" containerID="cri-o://471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3" gracePeriod=30 Sep 16 19:16:56.279131 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.279096 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kube-rbac-proxy" containerID="cri-o://b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977" gracePeriod=30 Sep 16 19:16:56.358960 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.358925 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:56.466578 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.466542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:56.520111 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.520084 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc"] Sep 16 19:16:56.520353 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.520341 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kube-rbac-proxy" Sep 16 19:16:56.520408 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.520355 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kube-rbac-proxy" Sep 16 19:16:56.520408 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.520368 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" Sep 16 19:16:56.520408 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.520373 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" Sep 16 19:16:56.520534 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.520421 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kube-rbac-proxy" Sep 16 19:16:56.520534 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.520430 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="30fd4c11-16ff-431d-8dbb-6d01582305b8" containerName="kserve-container" Sep 16 19:16:56.523223 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.523204 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.526635 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.526618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4a90a-predictor-serving-cert\"" Sep 16 19:16:56.526736 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.526674 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4a90a-kube-rbac-proxy-sar-config\"" Sep 16 19:16:56.538293 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.538267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc"] Sep 16 19:16:56.573925 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.573889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:56.683395 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.683310 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:16:56.687348 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.687322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.687469 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.687363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-proxy-tls\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.687469 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.687406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b7t6\" (UniqueName: \"kubernetes.io/projected/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-api-access-9b7t6\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.737646 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.737616 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a1c0844-99c2-4196-9069-33194ea2652e" containerID="b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977" exitCode=2 Sep 16 19:16:56.737765 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.737681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" event={"ID":"1a1c0844-99c2-4196-9069-33194ea2652e","Type":"ContainerDied","Data":"b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977"} Sep 16 19:16:56.788097 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.788062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b7t6\" (UniqueName: \"kubernetes.io/projected/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-api-access-9b7t6\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.788269 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.788112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.788269 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.788141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-proxy-tls\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.788860 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.788838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.790559 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.790540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-proxy-tls\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.797191 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.797169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b7t6\" (UniqueName: \"kubernetes.io/projected/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-api-access-9b7t6\") pod \"error-404-isvc-4a90a-predictor-6759885b-smchc\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:56.833186 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:56.833149 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:57.106932 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.106865 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc"] Sep 16 19:16:57.109453 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:16:57.109416 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ae2c1d_7a06_43ed_ae65_5b3f2a38ba11.slice/crio-74b463289094c5ef326606cc5f63052a867ebc5cd91703dcf1bfe91ea38d1817 WatchSource:0}: Error finding container 74b463289094c5ef326606cc5f63052a867ebc5cd91703dcf1bfe91ea38d1817: Status 404 returned error can't find the container with id 74b463289094c5ef326606cc5f63052a867ebc5cd91703dcf1bfe91ea38d1817 Sep 16 19:16:57.743145 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.743093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" event={"ID":"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11","Type":"ContainerStarted","Data":"059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8"} Sep 16 19:16:57.743552 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.743200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" event={"ID":"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11","Type":"ContainerStarted","Data":"c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4"} Sep 16 19:16:57.743552 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.743227 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" event={"ID":"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11","Type":"ContainerStarted","Data":"74b463289094c5ef326606cc5f63052a867ebc5cd91703dcf1bfe91ea38d1817"} Sep 16 19:16:57.743552 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.743250 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:57.743552 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.743273 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:16:57.744617 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.744590 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:16:57.762272 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:57.762218 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podStartSLOduration=1.762201794 podStartE2EDuration="1.762201794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:57.762136559 +0000 UTC m=+1955.408231377" watchObservedRunningTime="2026-09-16 19:16:57.762201794 +0000 UTC m=+1955.408296601" Sep 16 19:16:58.748757 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:58.748714 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:16:59.525321 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.525298 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:16:59.610840 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.610755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf2wt\" (UniqueName: \"kubernetes.io/projected/1a1c0844-99c2-4196-9069-33194ea2652e-kube-api-access-xf2wt\") pod \"1a1c0844-99c2-4196-9069-33194ea2652e\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " Sep 16 19:16:59.610840 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.610819 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0844-99c2-4196-9069-33194ea2652e-kube-rbac-proxy-sar-config\") pod \"1a1c0844-99c2-4196-9069-33194ea2652e\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " Sep 16 19:16:59.611052 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.610897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls\") pod \"1a1c0844-99c2-4196-9069-33194ea2652e\" (UID: \"1a1c0844-99c2-4196-9069-33194ea2652e\") " Sep 16 19:16:59.611131 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.611096 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1c0844-99c2-4196-9069-33194ea2652e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1a1c0844-99c2-4196-9069-33194ea2652e" (UID: "1a1c0844-99c2-4196-9069-33194ea2652e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:16:59.612936 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.612910 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1a1c0844-99c2-4196-9069-33194ea2652e" (UID: "1a1c0844-99c2-4196-9069-33194ea2652e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:16:59.613021 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.612996 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1c0844-99c2-4196-9069-33194ea2652e-kube-api-access-xf2wt" (OuterVolumeSpecName: "kube-api-access-xf2wt") pod "1a1c0844-99c2-4196-9069-33194ea2652e" (UID: "1a1c0844-99c2-4196-9069-33194ea2652e"). InnerVolumeSpecName "kube-api-access-xf2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:16:59.712238 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.712204 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1a1c0844-99c2-4196-9069-33194ea2652e-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:16:59.712238 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.712232 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf2wt\" (UniqueName: \"kubernetes.io/projected/1a1c0844-99c2-4196-9069-33194ea2652e-kube-api-access-xf2wt\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:16:59.712238 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.712241 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1a1c0844-99c2-4196-9069-33194ea2652e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:16:59.753954 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.753922 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a1c0844-99c2-4196-9069-33194ea2652e" containerID="471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3" exitCode=0 Sep 16 19:16:59.754372 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.753981 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" Sep 16 19:16:59.754372 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.754003 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" event={"ID":"1a1c0844-99c2-4196-9069-33194ea2652e","Type":"ContainerDied","Data":"471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3"} Sep 16 19:16:59.754372 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.754064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4" event={"ID":"1a1c0844-99c2-4196-9069-33194ea2652e","Type":"ContainerDied","Data":"dcb921818dda4278a2a58d3026c57e62797d9369b00cc155148588dfb53ea0b1"} Sep 16 19:16:59.754372 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.754091 2576 scope.go:117] "RemoveContainer" containerID="b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977" Sep 16 19:16:59.762596 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.762579 2576 scope.go:117] "RemoveContainer" containerID="471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3" Sep 16 19:16:59.769744 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.769727 2576 scope.go:117] "RemoveContainer" containerID="b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977" Sep 16 19:16:59.769991 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:16:59.769971 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977\": container with ID starting with b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977 not found: ID does not exist" containerID="b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977" Sep 16 19:16:59.770038 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.769998 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977"} err="failed to get container status \"b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977\": rpc error: code = NotFound desc = could not find container \"b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977\": container with ID starting with b523580d3e99e3033b0e7f738ddd2ce019838e948db7830d2b9e9c7a4fae8977 not found: ID does not exist" Sep 16 19:16:59.770038 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.770013 2576 scope.go:117] "RemoveContainer" containerID="471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3" Sep 16 19:16:59.770254 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:16:59.770238 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3\": container with ID starting with 471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3 not found: ID does not exist" containerID="471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3" Sep 16 19:16:59.770290 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.770261 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3"} err="failed to get container status \"471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3\": rpc error: code = NotFound desc = could not find container \"471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3\": container with ID starting with 471bcc9f502e742a4ce985e9656915e1ad9195a1f2ab7b244d3248dc5a1d2ec3 not found: ID does not exist" Sep 16 19:16:59.775690 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.775669 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4"] Sep 16 19:16:59.777861 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:16:59.777842 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-da3eb-predictor-5c7778cd96-zctq4"] Sep 16 19:17:00.884443 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:00.884411 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" path="/var/lib/kubelet/pods/1a1c0844-99c2-4196-9069-33194ea2652e/volumes" Sep 16 19:17:03.754446 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:03.754403 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:17:03.754900 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:03.754875 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:17:04.645414 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:04.645374 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:17:13.755466 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:13.755372 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:17:14.646636 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:14.646602 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:17:23.755362 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:23.755322 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:17:25.921337 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:25.921287 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:26.954338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:26.954300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:33.754892 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:33.754840 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:17:43.756201 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:43.756170 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:17:56.405682 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:56.405648 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:17:56.838301 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:17:56.838207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:19:22.911654 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:19:22.911627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:19:22.912779 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:19:22.912759 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:19:34.753934 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:19:34.753902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:20:39.867102 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:20:39.867022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:23:16.731876 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:23:16.731794 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:24:22.933970 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:24:22.933939 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:24:22.937058 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:24:22.937040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:26:11.022071 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.022034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:11.126167 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.126131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:11.132024 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.132000 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc"] Sep 16 19:26:11.132262 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.132233 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" containerID="cri-o://c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4" gracePeriod=30 Sep 16 19:26:11.132355 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.132261 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kube-rbac-proxy" containerID="cri-o://059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8" gracePeriod=30 Sep 16 19:26:11.188974 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.188928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:26:11.704982 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.704949 2576 generic.go:356] "Generic (PLEG): container finished" podID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerID="059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8" exitCode=2 Sep 16 19:26:11.705153 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:11.705025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" event={"ID":"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11","Type":"ContainerDied","Data":"059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8"} Sep 16 19:26:13.749471 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:13.749386 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 16 19:26:13.755848 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:13.755823 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:26:14.179521 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.179474 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:26:14.227724 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.227694 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-rbac-proxy-sar-config\") pod \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " Sep 16 19:26:14.227912 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.227741 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b7t6\" (UniqueName: \"kubernetes.io/projected/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-api-access-9b7t6\") pod \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " Sep 16 19:26:14.227912 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.227771 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-proxy-tls\") pod \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\" (UID: \"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11\") " Sep 16 19:26:14.228166 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.228132 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" (UID: "33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:26:14.230035 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.230009 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-api-access-9b7t6" (OuterVolumeSpecName: "kube-api-access-9b7t6") pod "33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" (UID: "33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11"). InnerVolumeSpecName "kube-api-access-9b7t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:26:14.230035 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.230020 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" (UID: "33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:26:14.328761 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.328673 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b7t6\" (UniqueName: \"kubernetes.io/projected/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-api-access-9b7t6\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:26:14.328761 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.328702 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:26:14.328761 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.328715 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:26:14.716973 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.716942 2576 generic.go:356] "Generic (PLEG): container finished" podID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerID="c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4" exitCode=0 Sep 16 19:26:14.717142 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.716988 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" Sep 16 19:26:14.717142 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.717004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" event={"ID":"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11","Type":"ContainerDied","Data":"c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4"} Sep 16 19:26:14.717142 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.717049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc" event={"ID":"33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11","Type":"ContainerDied","Data":"74b463289094c5ef326606cc5f63052a867ebc5cd91703dcf1bfe91ea38d1817"} Sep 16 19:26:14.717142 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.717073 2576 scope.go:117] "RemoveContainer" containerID="059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8" Sep 16 19:26:14.725242 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.725224 2576 scope.go:117] "RemoveContainer" containerID="c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4" Sep 16 19:26:14.732344 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.732330 2576 scope.go:117] "RemoveContainer" containerID="059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8" Sep 16 19:26:14.732703 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:26:14.732684 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8\": container with ID starting with 059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8 not found: ID does not exist" containerID="059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8" Sep 16 19:26:14.732762 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.732713 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8"} err="failed to get container status \"059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8\": rpc error: code = NotFound desc = could not find container \"059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8\": container with ID starting with 059bb3b2c54ff40c1744d6d495d3615cc7b4a8dccbdc9f8b24d641877aa727d8 not found: ID does not exist" Sep 16 19:26:14.732762 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.732729 2576 scope.go:117] "RemoveContainer" containerID="c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4" Sep 16 19:26:14.732967 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:26:14.732951 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4\": container with ID starting with c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4 not found: ID does not exist" containerID="c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4" Sep 16 19:26:14.733002 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.732970 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4"} err="failed to get container status \"c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4\": rpc error: code = NotFound desc = could not find container \"c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4\": container with ID starting with c92f83d0ed92653137b9286de8cac369600fa5ffce2624b8ca063e434fd9a4c4 not found: ID does not exist" Sep 16 19:26:14.737604 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.737583 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc"] Sep 16 19:26:14.741402 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.741380 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4a90a-predictor-6759885b-smchc"] Sep 16 19:26:14.886717 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:26:14.885095 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" path="/var/lib/kubelet/pods/33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11/volumes" Sep 16 19:29:18.168653 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:29:18.168580 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:29:22.959833 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:29:22.959795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:29:22.963741 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:29:22.963716 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:31:40.701855 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:31:40.701823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:45.250391 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:45.250308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:45.346185 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:45.346141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:45.365438 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:45.365407 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn"] Sep 16 19:33:45.365678 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:45.365652 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" containerID="cri-o://a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203" gracePeriod=30 Sep 16 19:33:45.365758 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:45.365679 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kube-rbac-proxy" containerID="cri-o://0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d" gracePeriod=30 Sep 16 19:33:45.443501 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:45.443447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:46.271536 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:46.271475 2576 generic.go:356] "Generic (PLEG): container finished" podID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerID="0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d" exitCode=2 Sep 16 19:33:46.271536 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:46.271512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" event={"ID":"655b1db8-2a90-41d2-9723-a2d8ed979c8c","Type":"ContainerDied","Data":"0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d"} Sep 16 19:33:47.061855 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:47.061821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:47.195758 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:47.195721 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:33:48.205789 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.205767 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:33:48.279804 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.279774 2576 generic.go:356] "Generic (PLEG): container finished" podID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerID="a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203" exitCode=0 Sep 16 19:33:48.279976 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.279816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" event={"ID":"655b1db8-2a90-41d2-9723-a2d8ed979c8c","Type":"ContainerDied","Data":"a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203"} Sep 16 19:33:48.279976 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.279828 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" Sep 16 19:33:48.279976 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.279864 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn" event={"ID":"655b1db8-2a90-41d2-9723-a2d8ed979c8c","Type":"ContainerDied","Data":"46e04f0afc900c1dcb082ec2919e3a7519b1b47abbf6ad26a0ef6fc8601cb798"} Sep 16 19:33:48.279976 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.279893 2576 scope.go:117] "RemoveContainer" containerID="0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d" Sep 16 19:33:48.287874 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.287856 2576 scope.go:117] "RemoveContainer" containerID="a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203" Sep 16 19:33:48.295260 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.295243 2576 scope.go:117] "RemoveContainer" containerID="0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d" Sep 16 19:33:48.295550 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:33:48.295526 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d\": container with ID starting with 0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d not found: ID does not exist" containerID="0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d" Sep 16 19:33:48.295648 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.295563 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d"} err="failed to get container status \"0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d\": rpc error: code = NotFound desc = could not find container \"0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d\": container with ID starting with 0959f2349283ccc0e626ce819c06d2b11ea39cce7a42fdb5917806dd542aa44d not found: ID does not exist" Sep 16 19:33:48.295648 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.295588 2576 scope.go:117] "RemoveContainer" containerID="a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203" Sep 16 19:33:48.295922 ip-10-0-138-79 kubenswrapper[2576]: E0916 19:33:48.295902 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203\": container with ID starting with a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203 not found: ID does not exist" containerID="a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203" Sep 16 19:33:48.296018 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.295926 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203"} err="failed to get container status \"a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203\": rpc error: code = NotFound desc = could not find container \"a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203\": container with ID starting with a86e0bfcf327c28b50d015051f0da5536b8ce73df7def20bb415a868c0769203 not found: ID does not exist" Sep 16 19:33:48.315238 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.315180 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szh46\" (UniqueName: \"kubernetes.io/projected/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-api-access-szh46\") pod \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " Sep 16 19:33:48.315238 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.315219 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls\") pod \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " Sep 16 19:33:48.315414 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.315281 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-rbac-proxy-sar-config\") pod \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\" (UID: \"655b1db8-2a90-41d2-9723-a2d8ed979c8c\") " Sep 16 19:33:48.315684 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.315660 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "655b1db8-2a90-41d2-9723-a2d8ed979c8c" (UID: "655b1db8-2a90-41d2-9723-a2d8ed979c8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:33:48.317325 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.317303 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "655b1db8-2a90-41d2-9723-a2d8ed979c8c" (UID: "655b1db8-2a90-41d2-9723-a2d8ed979c8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:33:48.317427 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.317404 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-api-access-szh46" (OuterVolumeSpecName: "kube-api-access-szh46") pod "655b1db8-2a90-41d2-9723-a2d8ed979c8c" (UID: "655b1db8-2a90-41d2-9723-a2d8ed979c8c"). InnerVolumeSpecName "kube-api-access-szh46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:33:48.416215 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.416180 2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/655b1db8-2a90-41d2-9723-a2d8ed979c8c-proxy-tls\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:33:48.416215 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.416206 2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:33:48.416215 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.416215 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szh46\" (UniqueName: \"kubernetes.io/projected/655b1db8-2a90-41d2-9723-a2d8ed979c8c-kube-api-access-szh46\") on node \"ip-10-0-138-79.ec2.internal\" DevicePath \"\"" Sep 16 19:33:48.630785 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.630752 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn"] Sep 16 19:33:48.633364 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.633341 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-ca22e-predictor-6b8dc9599b-mjzpn"] Sep 16 19:33:48.882615 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:33:48.882581 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" path="/var/lib/kubelet/pods/655b1db8-2a90-41d2-9723-a2d8ed979c8c/volumes" Sep 16 19:34:18.121052 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:18.121017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:18.294498 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:18.294453 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:18.371287 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:18.371203 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:21.177526 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:21.177477 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sm66h_f2e9015b-775b-4a7d-8542-9a1222e7f67b/global-pull-secret-syncer/0.log" Sep 16 19:34:21.264461 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:21.264434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" Sep 16 19:34:21.438104 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:21.437998 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-79.ec2.internal_250437f78067fb73286aae1b34e08bfc/haproxy/0.log" Sep 16 19:34:22.979781 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:22.979751 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:34:22.984238 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:22.984218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:34:24.818824 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:24.818790 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:26.037477 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:26.037434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j7vb_a8dd90d6-b781-42af-8aa8-2812cd343c88/node-exporter/0.log" Sep 16 19:34:26.085001 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:26.084962 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j7vb_a8dd90d6-b781-42af-8aa8-2812cd343c88/kube-rbac-proxy/0.log" Sep 16 19:34:26.126209 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:26.126182 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7j7vb_a8dd90d6-b781-42af-8aa8-2812cd343c88/init-textfile/0.log" Sep 16 19:34:27.666515 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:27.666471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-htfgb_4b647a36-5a97-423e-af52-e5cf39fa9a1d/konnectivity-agent/0.log" limit=1024 Sep 16 19:34:29.208983 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.208952 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt"] Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209193 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kube-rbac-proxy" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209203 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kube-rbac-proxy" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209215 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kube-rbac-proxy" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209220 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kube-rbac-proxy" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209229 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209234 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209244 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209252 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209260 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kube-rbac-proxy" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209267 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kube-rbac-proxy" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209276 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209282 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209325 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209331 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209338 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kserve-container" Sep 16 19:34:29.209338 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209343 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="655b1db8-2a90-41d2-9723-a2d8ed979c8c" containerName="kube-rbac-proxy" Sep 16 19:34:29.209869 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209349 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="33ae2c1d-7a06-43ed-ae65-5b3f2a38ba11" containerName="kube-rbac-proxy" Sep 16 19:34:29.209869 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.209354 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a1c0844-99c2-4196-9069-33194ea2652e" containerName="kube-rbac-proxy" Sep 16 19:34:29.213014 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.212995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.215876 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.215858 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lpmdv\"/\"default-dockercfg-87trx\"" Sep 16 19:34:29.216689 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.216670 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpmdv\"/\"kube-root-ca.crt\"" Sep 16 19:34:29.216791 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.216708 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lpmdv\"/\"openshift-service-ca.crt\"" Sep 16 19:34:29.223249 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.223229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt"] Sep 16 19:34:29.300721 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.300687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-lib-modules\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.300912 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.300752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-podres\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.300912 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.300809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-sys\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.300912 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.300830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-proc\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.300912 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.300851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-699qw\" (UniqueName: \"kubernetes.io/projected/1f934ad8-2313-4e26-85d5-a92e0703050f-kube-api-access-699qw\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401348 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401288 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-proc\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401348 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401334 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-699qw\" (UniqueName: \"kubernetes.io/projected/1f934ad8-2313-4e26-85d5-a92e0703050f-kube-api-access-699qw\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401706 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-lib-modules\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401706 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401410 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-proc\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401706 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-podres\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401706 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-sys\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401706 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401589 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-lib-modules\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401706 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-sys\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.401706 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.401623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f934ad8-2313-4e26-85d5-a92e0703050f-podres\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.410993 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.410970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-699qw\" (UniqueName: \"kubernetes.io/projected/1f934ad8-2313-4e26-85d5-a92e0703050f-kube-api-access-699qw\") pod \"perf-node-gather-daemonset-np8nt\" (UID: \"1f934ad8-2313-4e26-85d5-a92e0703050f\") " pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.522659 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.522573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:29.821168 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:29.821092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt"] Sep 16 19:34:29.824305 ip-10-0-138-79 kubenswrapper[2576]: W0916 19:34:29.824276 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1f934ad8_2313_4e26_85d5_a92e0703050f.slice/crio-98bef43fa6322137d47f6a7f33b3cbe92eaad80ce64f593e24dfd38fe7b967ea WatchSource:0}: Error finding container 98bef43fa6322137d47f6a7f33b3cbe92eaad80ce64f593e24dfd38fe7b967ea: Status 404 returned error can't find the container with id 98bef43fa6322137d47f6a7f33b3cbe92eaad80ce64f593e24dfd38fe7b967ea Sep 16 19:34:30.426631 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:30.426597 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" event={"ID":"1f934ad8-2313-4e26-85d5-a92e0703050f","Type":"ContainerStarted","Data":"37fc100a698de714f7710a1aaf164ba953bd5cb2991f4d07969652bcfe5e87b3"} Sep 16 19:34:30.427001 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:30.426657 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" event={"ID":"1f934ad8-2313-4e26-85d5-a92e0703050f","Type":"ContainerStarted","Data":"98bef43fa6322137d47f6a7f33b3cbe92eaad80ce64f593e24dfd38fe7b967ea"} Sep 16 19:34:30.427001 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:30.426731 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:30.445209 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:30.445163 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" podStartSLOduration=1.4451299309999999 podStartE2EDuration="1.445129931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:30.443722982 +0000 UTC m=+3008.089817787" watchObservedRunningTime="2026-09-16 19:34:30.445129931 +0000 UTC m=+3008.091224758" Sep 16 19:34:30.925906 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:30.925883 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8c758_77417086-a1e1-4b6a-b8fe-b168e3c07c59/dns/0.log" Sep 16 19:34:30.953400 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:30.953381 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8c758_77417086-a1e1-4b6a-b8fe-b168e3c07c59/kube-rbac-proxy/0.log" Sep 16 19:34:31.150536 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:31.150500 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4p8g_2e46d10a-d3c7-4b28-bab1-a16ffbe596b1/dns-node-resolver/0.log" Sep 16 19:34:31.690054 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:31.690023 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rcnm2_23954e0d-2798-4c79-8b43-244302141d00/node-ca/0.log" Sep 16 19:34:32.858917 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:32.858888 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6fbqp_735e38d3-3525-407d-9e6c-06aedad02c7a/serve-healthcheck-canary/0.log" Sep 16 19:34:33.465391 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:33.465338 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hcq9_4243836b-ee23-4d88-ae22-04619428f84d/kube-rbac-proxy/0.log" Sep 16 19:34:33.503270 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:33.503250 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hcq9_4243836b-ee23-4d88-ae22-04619428f84d/exporter/0.log" Sep 16 19:34:33.530369 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:33.530340 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hcq9_4243836b-ee23-4d88-ae22-04619428f84d/extractor/0.log" Sep 16 19:34:36.197261 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:36.197237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-f8c46b677-w98w5_f982db09-27cd-4427-bff1-ac58a9a9dea5/manager/0.log" Sep 16 19:34:36.439372 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:36.439346 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lpmdv/perf-node-gather-daemonset-np8nt" Sep 16 19:34:36.505139 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:36.505062 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-bcld6_36905445-43b6-4adb-8fc0-d754b3aa6a63/seaweedfs/0.log" Sep 16 19:34:42.342919 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.342893 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldjt8_5f973677-db95-4065-851a-c26414a4c960/kube-multus-additional-cni-plugins/0.log" Sep 16 19:34:42.367956 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.367926 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldjt8_5f973677-db95-4065-851a-c26414a4c960/egress-router-binary-copy/0.log" Sep 16 19:34:42.394832 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.394806 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldjt8_5f973677-db95-4065-851a-c26414a4c960/cni-plugins/0.log" Sep 16 19:34:42.416725 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.416705 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldjt8_5f973677-db95-4065-851a-c26414a4c960/bond-cni-plugin/0.log" Sep 16 19:34:42.438869 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.438850 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldjt8_5f973677-db95-4065-851a-c26414a4c960/routeoverride-cni/0.log" Sep 16 19:34:42.462669 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.462645 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldjt8_5f973677-db95-4065-851a-c26414a4c960/whereabouts-cni-bincopy/0.log" Sep 16 19:34:42.486365 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.486348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldjt8_5f973677-db95-4065-851a-c26414a4c960/whereabouts-cni/0.log" Sep 16 19:34:42.865966 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.865937 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dwbzs_80653a37-6288-47de-8d79-e913d7a23687/kube-multus/0.log" Sep 16 19:34:42.997352 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:42.997322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b47hq_e3fd2611-8884-4a84-861e-1342cee39e99/network-metrics-daemon/0.log" Sep 16 19:34:43.047672 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:43.047632 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-b47hq_e3fd2611-8884-4a84-861e-1342cee39e99/kube-rbac-proxy/0.log" Sep 16 19:34:44.299908 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.299875 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-controller/0.log" Sep 16 19:34:44.320440 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.320417 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/1.log" Sep 16 19:34:44.320592 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.320457 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovn-acl-logging/0.log" Sep 16 19:34:44.339208 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.339185 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/kube-rbac-proxy-node/0.log" Sep 16 19:34:44.365761 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.365743 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 19:34:44.386388 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.386369 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/northd/0.log" Sep 16 19:34:44.409048 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.409021 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/nbdb/0.log" Sep 16 19:34:44.431010 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.430993 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/sbdb/0.log" Sep 16 19:34:44.469357 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:44.469323 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xx2wj_c6d71861-fc83-4426-9f3e-15ca25b91f00/ovnkube-controller/0.log" Sep 16 19:34:45.486445 ip-10-0-138-79 kubenswrapper[2576]: I0916 19:34:45.486418 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-w8b6s_6b6ef13b-8221-4c89-981c-d17ed779bbca/network-check-target-container/0.log"