Sep 17 17:40:53.566834 ip-10-0-135-40 systemd[1]: Starting Kubernetes Kubelet... Sep 17 17:40:53.995319 ip-10-0-135-40 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 17:40:53.995319 ip-10-0-135-40 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 17:40:53.995319 ip-10-0-135-40 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 17:40:53.995319 ip-10-0-135-40 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 17:40:53.995319 ip-10-0-135-40 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 17:40:53.996335 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:53.996241 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 17:40:54.000835 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000815 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000840 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000846 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000850 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000854 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000859 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000863 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000867 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000881 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:40:54.000883 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000886 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000890 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000895 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000898 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000902 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000906 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000910 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000915 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000919 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000923 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000927 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000931 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000934 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000938 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000942 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000946 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000949 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000953 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000959 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000963 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:40:54.001143 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000966 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000970 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000974 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000978 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000981 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000985 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000989 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000993 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.000996 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001001 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001005 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001009 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001012 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001017 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001022 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001026 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001030 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001034 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001040 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001043 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:40:54.001656 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001046 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001050 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001054 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001058 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001062 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001066 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001070 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001074 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001077 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001081 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001085 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001091 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001094 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001099 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001103 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001107 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001111 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001115 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001118 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001123 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:40:54.002173 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001129 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001134 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001139 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001143 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001147 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001154 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001159 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001164 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001169 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001173 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001177 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001182 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001186 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001190 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001194 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001199 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001205 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001209 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001213 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:40:54.002687 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001217 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001221 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001225 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001230 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001234 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001238 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001243 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001247 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001846 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001854 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:40:54.003148 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.001858 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:40:54.003628 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003614 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003629 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003635 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003639 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003643 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003648 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003651 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003655 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003659 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003663 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:40:54.003665 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003667 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003672 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003676 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003680 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003683 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003687 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003691 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003695 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003698 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003702 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003706 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003710 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003714 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003717 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003721 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003725 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003729 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003734 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003737 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003741 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:40:54.003923 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003745 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003748 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003752 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003756 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003760 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003764 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003769 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003774 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003777 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003781 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003784 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003788 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003792 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003795 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003800 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003805 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003809 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003813 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003816 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:40:54.004548 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003820 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003824 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003828 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003831 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003838 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003843 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003856 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003861 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003865 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003869 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003873 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003879 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003884 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003888 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003892 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003896 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003900 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003905 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003910 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:40:54.005022 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003914 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003919 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003924 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003929 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003933 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003937 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003942 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003947 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003952 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003958 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003963 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003967 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003971 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003975 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003978 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003981 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003985 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003989 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003993 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.003996 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:40:54.005563 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004000 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004005 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004008 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004012 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004016 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004021 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004025 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004028 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004032 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004036 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004040 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004631 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004639 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004643 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004647 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004650 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004655 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004659 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004663 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004667 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:40:54.006047 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004671 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004675 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004679 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004684 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004688 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004691 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004695 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004699 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004703 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004706 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004710 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004713 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004718 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004722 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004727 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004732 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004737 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004742 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004746 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:40:54.006538 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004750 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004754 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004758 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004761 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004766 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004770 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004774 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004778 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004782 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004786 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004790 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004793 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004797 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004801 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004804 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004808 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004811 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004815 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004819 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004823 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:40:54.007053 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004826 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004830 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004834 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004838 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004841 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004845 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004850 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004853 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004856 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004860 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004865 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004869 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004873 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004876 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004880 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004884 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004890 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004896 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004900 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004905 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:40:54.007553 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004909 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004913 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004918 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004924 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004928 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004933 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004938 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004942 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004946 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004949 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004953 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004957 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004961 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004965 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004968 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004974 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004978 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004982 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004985 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:40:54.008058 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004989 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004993 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.004998 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005001 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005005 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005009 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005013 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005017 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005020 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005024 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005029 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.005033 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005136 2550 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005182 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005192 2550 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005199 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005205 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005210 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005215 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005222 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005226 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005230 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 17:40:54.008569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005236 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005241 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005245 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005249 2550 flags.go:64] FLAG: --cgroup-root="" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005254 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005258 2550 flags.go:64] FLAG: --client-ca-file="" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005263 2550 flags.go:64] FLAG: --cloud-provider="external" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005267 2550 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005274 2550 flags.go:64] FLAG: --cluster-domain="" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005278 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005284 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005288 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005294 2550 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005300 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005305 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005309 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005315 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005321 2550 flags.go:64] FLAG: --contention-profiling="false" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005325 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005329 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005334 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005338 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005344 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005349 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005353 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 17:40:54.009118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005357 2550 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005361 2550 flags.go:64] FLAG: --enable-server="true" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005366 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005372 2550 flags.go:64] FLAG: --event-burst="100" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005376 2550 flags.go:64] FLAG: --event-qps="50" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005381 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005385 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005389 2550 flags.go:64] FLAG: --eviction-hard="" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005395 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005400 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005405 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005410 2550 flags.go:64] FLAG: --eviction-soft="" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005415 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005419 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005424 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005428 2550 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005433 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005437 2550 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005441 2550 flags.go:64] FLAG: --feature-gates="" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005447 2550 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005452 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005456 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005461 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005466 2550 flags.go:64] FLAG: --healthz-port="10248" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005471 2550 flags.go:64] FLAG: --help="false" Sep 17 17:40:54.009749 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005476 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005481 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005485 2550 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005490 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005495 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005500 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005504 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005509 2550 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005513 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005518 2550 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005522 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005540 2550 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005545 2550 flags.go:64] FLAG: --kube-reserved="" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005549 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005554 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005559 2550 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005563 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005568 2550 flags.go:64] FLAG: --lock-file="" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005572 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005577 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005582 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005593 2550 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005598 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005602 2550 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 17:40:54.010376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005607 2550 flags.go:64] FLAG: --logging-format="text" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005611 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005617 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005621 2550 flags.go:64] FLAG: --manifest-url="" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005626 2550 flags.go:64] FLAG: --manifest-url-header="" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005632 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005637 2550 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005643 2550 flags.go:64] FLAG: --max-pods="110" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005649 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005654 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005658 2550 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005662 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005667 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005671 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005676 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005686 2550 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005691 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005696 2550 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005700 2550 flags.go:64] FLAG: --pod-cidr="" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005704 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005713 2550 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005717 2550 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005722 2550 flags.go:64] FLAG: --pods-per-core="0" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005726 2550 flags.go:64] FLAG: --port="10250" Sep 17 17:40:54.011011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005731 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005735 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0866d935d61da4091" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005740 2550 flags.go:64] FLAG: --qos-reserved="" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005744 2550 flags.go:64] FLAG: --read-only-port="10255" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005749 2550 flags.go:64] FLAG: --register-node="true" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005753 2550 flags.go:64] FLAG: --register-with-taints="" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005760 2550 flags.go:64] FLAG: --registry-burst="10" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005765 2550 flags.go:64] FLAG: --registry-qps="5" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005769 2550 flags.go:64] FLAG: --reserved-cpus="" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005775 2550 flags.go:64] FLAG: --reserved-memory="" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005781 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005786 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005790 2550 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005795 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005799 2550 flags.go:64] FLAG: --runonce="false" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005803 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005808 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005814 2550 flags.go:64] FLAG: --seccomp-default="false" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005819 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005832 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005837 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005841 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005846 2550 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005850 2550 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005855 2550 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 17:40:54.011628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005860 2550 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005864 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005868 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005872 2550 flags.go:64] FLAG: --system-cgroups="" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005877 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005884 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005889 2550 flags.go:64] FLAG: --tls-cert-file="" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005893 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005899 2550 flags.go:64] FLAG: --tls-min-version="" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005903 2550 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005908 2550 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005912 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005916 2550 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005922 2550 flags.go:64] FLAG: --v="2" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005929 2550 flags.go:64] FLAG: --version="false" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005935 2550 flags.go:64] FLAG: --vmodule="" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005943 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.005948 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006112 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006118 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006122 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006126 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006130 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006134 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:40:54.012275 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006137 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006143 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006147 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006151 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006155 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006159 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006162 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006167 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006170 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006174 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006178 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006182 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006186 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006190 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006193 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006198 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006201 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006205 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006208 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006212 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:40:54.012902 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006216 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006221 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006225 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006229 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006235 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006239 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006243 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006247 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006251 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006255 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006259 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006263 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006267 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006271 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006275 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006279 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006284 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006287 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006291 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:40:54.013383 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006295 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006302 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006307 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006312 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006317 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006321 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006325 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006329 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006333 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006336 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006340 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006344 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006348 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006351 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006357 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006361 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006365 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006371 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006375 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:40:54.014140 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006379 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006383 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006387 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006391 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006395 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006399 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006403 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006408 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006413 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006418 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006422 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006426 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006430 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006433 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006437 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006441 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006445 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006448 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006452 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006456 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:40:54.014777 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006460 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006464 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006467 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006471 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006475 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006479 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006483 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006488 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006494 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006499 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006506 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006510 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006514 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006519 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.006523 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:40:54.015274 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.007031 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 17:40:54.015884 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.015859 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 17:40:54.015884 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.015884 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 17:40:54.016694 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.016681 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 17:40:54.016733 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.016695 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 17:40:54.016769 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016760 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:40:54.016769 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016768 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016772 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016776 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016779 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016783 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016788 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016791 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016794 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016797 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016800 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016803 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016806 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016808 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016811 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016814 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016817 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016820 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016823 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016826 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:40:54.016820 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016828 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016832 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016836 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016839 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016843 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016846 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016850 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016854 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016856 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016859 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016862 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016865 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016868 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016870 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016873 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016875 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016878 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016881 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016884 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:40:54.017281 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016888 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016891 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016894 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016897 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016900 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016902 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016905 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016908 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016911 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016914 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016917 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016919 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016922 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016925 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016928 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016932 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016935 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016938 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016941 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016943 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:40:54.017829 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016946 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016948 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016951 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016954 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016956 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016959 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016962 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016965 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016967 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016970 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016973 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016975 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016978 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016980 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016983 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016986 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016988 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016991 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016994 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.016996 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:40:54.018314 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017000 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017003 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017006 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017009 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017012 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017015 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017018 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017021 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017042 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017046 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017049 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017052 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017056 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017058 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017061 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017064 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017067 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017070 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017072 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:40:54.018816 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017075 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.017080 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017199 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017205 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017207 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017210 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017214 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017217 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017220 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017223 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017226 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017228 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017231 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017233 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017237 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 17:40:54.019285 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017239 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017242 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017245 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017248 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017252 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017257 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017260 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017264 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017268 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017271 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017273 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017276 2550 feature_gate.go:328] unrecognized feature gate: Example Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017279 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017281 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017284 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017287 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017290 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017293 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017295 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 17:40:54.019672 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017298 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017301 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017303 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017306 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017308 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017311 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017313 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017316 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017318 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017321 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017324 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017327 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017330 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017333 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017336 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017338 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017341 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017344 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017347 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 17:40:54.020131 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017349 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017352 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017355 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017357 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017360 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017363 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017365 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017368 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017371 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017373 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017376 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017379 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017382 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017386 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017389 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017392 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017394 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017397 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017399 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017402 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 17:40:54.020603 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017404 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017407 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017409 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017412 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017415 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017418 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017420 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017424 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017427 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017429 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017432 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017435 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017437 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017440 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017443 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017446 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017449 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017451 2550 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017454 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017456 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 17:40:54.021073 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017459 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017462 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017464 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017466 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017469 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017472 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017475 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:54.017477 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.017482 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.017636 2550 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 17:40:54.021662 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.020237 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 17:40:54.021909 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.021682 2550 server.go:1019] "Starting client certificate rotation" Sep 17 17:40:54.021909 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.021787 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 17:40:54.021909 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.021825 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 17:40:54.024438 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.024424 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 17:40:54.025663 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.025640 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 17:40:54.035768 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.035740 2550 log.go:25] "Validated CRI v1 runtime API" Sep 17 17:40:54.039517 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.039498 2550 log.go:25] "Validated CRI v1 image API" Sep 17 17:40:54.040492 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.040475 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 17:40:54.044665 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.044642 2550 fs.go:135] Filesystem UUIDs: map[3415556f-1870-497e-a09d-d3b47f8f36ee:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e756bb5a-2c67-4be4-972b-400ed13a3f10:/dev/nvme0n1p4] Sep 17 17:40:54.044665 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.044662 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 17:40:54.049314 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.049292 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 17:40:54.051329 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.051214 2550 manager.go:216] Machine: {Timestamp:2026-09-17 17:40:54.0500259 +0000 UTC m=+0.350593435 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099031 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e222af0c4ce2484c3ea3e2a28ef6c SystemUUID:ec2e222a-f0c4-ce24-84c3-ea3e2a28ef6c BootID:7ee02d01-a885-4e83-b04c-f2667b90ea28 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f5:4e:d2:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f5:4e:d2:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:15:74:b5:dd:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 17:40:54.051329 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.051317 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 17:40:54.051510 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.051417 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 17:40:54.052443 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.052417 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 17:40:54.052595 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.052445 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 17:40:54.053211 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.053200 2550 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 17:40:54.053244 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.053212 2550 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 17:40:54.053244 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.053220 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 17:40:54.053892 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.053879 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 17:40:54.053976 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.053961 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 17:40:54.057935 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.057918 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 17 17:40:54.058062 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058051 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 17:40:54.058104 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058085 2550 kubelet.go:486] "Attempting to sync node with API server" Sep 17 17:40:54.058104 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058098 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 17:40:54.058161 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058111 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 17:40:54.058161 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058123 2550 kubelet.go:398] "Adding apiserver pod source" Sep 17 17:40:54.058161 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058131 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 17:40:54.058570 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058547 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 17:40:54.058651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.058577 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 17:40:54.060934 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.060914 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 17 17:40:54.063086 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.063073 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 17:40:54.063142 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.063103 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 17:40:54.064648 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064635 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 17:40:54.064684 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064654 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 17:40:54.064684 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064659 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 17:40:54.064684 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064665 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 17:40:54.064684 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064673 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 17:40:54.064684 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064678 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 17:40:54.064819 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064691 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 17:40:54.064819 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064697 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 17:40:54.064819 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064704 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 17:40:54.064819 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064709 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 17:40:54.064819 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064718 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 17:40:54.064819 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.064724 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 17:40:54.066258 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.066245 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 17:40:54.066294 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.066259 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 17:40:54.068115 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.068088 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 17:40:54.068605 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.068566 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 17:40:54.070314 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.070299 2550 server.go:1268] "Started kubelet" Sep 17 17:40:54.070387 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.070367 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 17:40:54.070455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.070381 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 17:40:54.070455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.070416 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 17:40:54.070567 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.070460 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 17:40:54.071379 ip-10-0-135-40 systemd[1]: Started Kubernetes Kubelet. Sep 17 17:40:54.071706 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.071687 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 17:40:54.073166 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.073148 2550 server.go:310] "Adding debug handlers to kubelet server" Sep 17 17:40:54.073332 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.073313 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t7xtv" Sep 17 17:40:54.075051 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.075030 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 17:40:54.078316 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.078295 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 17:40:54.078316 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.078307 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 17:40:54.078455 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.078406 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.078898 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.078882 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 17:40:54.078981 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.078958 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 17:40:54.078981 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.078976 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 17:40:54.079142 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.079120 2550 reconstruct.go:97] "Volume reconstruction finished" Sep 17 17:40:54.079142 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.079134 2550 reconciler.go:29] "Reconciler: start to sync state" Sep 17 17:40:54.079228 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.079177 2550 factory.go:55] Registering systemd factory Sep 17 17:40:54.079228 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.079218 2550 factory.go:223] Registration of the systemd container factory successfully Sep 17 17:40:54.079457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.079443 2550 factory.go:153] Registering CRI-O factory Sep 17 17:40:54.079518 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.079461 2550 factory.go:223] Registration of the crio container factory successfully Sep 17 17:40:54.080401 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.080368 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 17:40:54.080475 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.080412 2550 factory.go:103] Registering Raw factory Sep 17 17:40:54.080475 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.080431 2550 manager.go:1196] Started watching for new ooms in manager Sep 17 17:40:54.080893 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.080877 2550 manager.go:349] Starting recovery of all containers Sep 17 17:40:54.081345 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.081309 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-t7xtv" Sep 17 17:40:54.082285 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.080822 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-40.ec2.internal.18d62c858f264a8c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-40.ec2.internal,UID:ip-10-0-135-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-40.ec2.internal,},FirstTimestamp:2026-09-17 17:40:54.0702747 +0000 UTC m=+0.370842235,LastTimestamp:2026-09-17 17:40:54.0702747 +0000 UTC m=+0.370842235,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-40.ec2.internal,}" Sep 17 17:40:54.089473 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.089423 2550 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 17:40:54.089780 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.089745 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 17:40:54.092820 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.092700 2550 manager.go:354] Recovery completed Sep 17 17:40:54.094550 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.094503 2550 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 17 17:40:54.097574 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.097561 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:40:54.099619 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.099523 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:40:54.099683 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.099641 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:40:54.099683 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.099659 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:40:54.100338 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.100324 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 17:40:54.100338 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.100332 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:40:54.100438 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.100338 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 17:40:54.100438 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.100398 2550 state_mem.go:36] "Initialized new in-memory state store" Sep 17 17:40:54.101219 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.101204 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:40:54.101302 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.101236 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:40:54.101302 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.101253 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:40:54.102466 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.102453 2550 policy_none.go:49] "None policy: Start" Sep 17 17:40:54.102517 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.102469 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 17:40:54.102517 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.102479 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 17:40:54.104396 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.104384 2550 policy_none.go:47] "Start" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.138629 2550 manager.go:337] "Starting Device Plugin manager" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.138669 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.138680 2550 server.go:87] "Starting device plugin registration server" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.138998 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.139008 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.139111 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.139178 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.139184 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.139790 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 17:40:54.154064 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.139832 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.207978 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.207937 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 17:40:54.209255 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.209236 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 17:40:54.209321 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.209294 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 17:40:54.209372 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.209321 2550 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 17:40:54.209447 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.209428 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 17:40:54.211346 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.211323 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:40:54.240099 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.240068 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:40:54.241063 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.241046 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:40:54.241158 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.241080 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:40:54.241158 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.241096 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:40:54.241158 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.241128 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.249102 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.249016 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.249102 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.249042 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-40.ec2.internal\": node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.263810 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.263782 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.310412 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.310383 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal"] Sep 17 17:40:54.310487 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.310462 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:40:54.312022 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.312004 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:40:54.312112 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.312039 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:40:54.312112 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.312052 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:40:54.314365 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.314352 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:40:54.314487 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.314470 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.314560 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.314511 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:40:54.315184 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.315160 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:40:54.315184 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.315188 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:40:54.315308 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.315198 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:40:54.315308 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.315166 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:40:54.315308 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.315262 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:40:54.315308 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.315274 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:40:54.317576 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.317559 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.317635 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.317593 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 17:40:54.318336 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.318322 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientMemory" Sep 17 17:40:54.318413 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.318347 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 17:40:54.318413 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.318360 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeHasSufficientPID" Sep 17 17:40:54.347480 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.347454 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-40.ec2.internal\" not found" node="ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.352098 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.352079 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-40.ec2.internal\" not found" node="ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.364245 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.364202 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.381692 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.381660 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf772d8abc581fe5ccec09571c5d2950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"bf772d8abc581fe5ccec09571c5d2950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.381813 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.381694 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf772d8abc581fe5ccec09571c5d2950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"bf772d8abc581fe5ccec09571c5d2950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.381813 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.381712 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/323a5990fa2f22ac49cb118e6b347864-config\") pod \"kube-apiserver-proxy-ip-10-0-135-40.ec2.internal\" (UID: \"323a5990fa2f22ac49cb118e6b347864\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.465369 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.465323 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.482694 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.482669 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf772d8abc581fe5ccec09571c5d2950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"bf772d8abc581fe5ccec09571c5d2950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.482774 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.482698 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf772d8abc581fe5ccec09571c5d2950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"bf772d8abc581fe5ccec09571c5d2950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.482774 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.482718 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/323a5990fa2f22ac49cb118e6b347864-config\") pod \"kube-apiserver-proxy-ip-10-0-135-40.ec2.internal\" (UID: \"323a5990fa2f22ac49cb118e6b347864\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.482841 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.482794 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/323a5990fa2f22ac49cb118e6b347864-config\") pod \"kube-apiserver-proxy-ip-10-0-135-40.ec2.internal\" (UID: \"323a5990fa2f22ac49cb118e6b347864\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.482841 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.482796 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bf772d8abc581fe5ccec09571c5d2950-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"bf772d8abc581fe5ccec09571c5d2950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.482841 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.482796 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bf772d8abc581fe5ccec09571c5d2950-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal\" (UID: \"bf772d8abc581fe5ccec09571c5d2950\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.566184 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.566112 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.651644 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.651606 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.655408 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:54.655386 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 17 17:40:54.666676 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.666638 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.767290 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.767246 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.867819 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.867739 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:54.968280 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:54.968243 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:55.020758 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.020719 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 17:40:55.021258 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.020868 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 17:40:55.040546 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.040500 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:40:55.068942 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:55.068907 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:55.088279 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.088242 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:35:54 +0000 UTC" deadline="2028-05-19 20:47:18.744702476 +0000 UTC" Sep 17 17:40:55.088279 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.088275 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14643h6m23.65643088s" Sep 17 17:40:55.169624 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:55.169590 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:55.270401 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:55.270356 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:55.306964 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:55.306933 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf772d8abc581fe5ccec09571c5d2950.slice/crio-5bae791eec3c7863e0a12c71e9f71bc9dfd075246d2f6f7b446a467e01ce58cf WatchSource:0}: Error finding container 5bae791eec3c7863e0a12c71e9f71bc9dfd075246d2f6f7b446a467e01ce58cf: Status 404 returned error can't find the container with id 5bae791eec3c7863e0a12c71e9f71bc9dfd075246d2f6f7b446a467e01ce58cf Sep 17 17:40:55.307235 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:55.307215 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323a5990fa2f22ac49cb118e6b347864.slice/crio-bd97465f31d95682e4d7f517b1c2f60a1b321c8b2a0822361862ca43e4cab0d2 WatchSource:0}: Error finding container bd97465f31d95682e4d7f517b1c2f60a1b321c8b2a0822361862ca43e4cab0d2: Status 404 returned error can't find the container with id bd97465f31d95682e4d7f517b1c2f60a1b321c8b2a0822361862ca43e4cab0d2 Sep 17 17:40:55.312681 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.312658 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 17:40:55.370502 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:55.370442 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:55.442592 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.442487 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:40:55.470933 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:55.470903 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:55.571779 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:55.571743 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-40.ec2.internal\" not found" Sep 17 17:40:55.664984 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.664954 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:40:55.678953 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.678927 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" Sep 17 17:40:55.691829 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.691799 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 17:40:55.692808 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.692766 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" Sep 17 17:40:55.700420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:55.700395 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 17:40:56.057624 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.057544 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 17:40:56.059555 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.059519 2550 apiserver.go:52] "Watching apiserver" Sep 17 17:40:56.067936 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.067902 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 17:40:56.068490 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.068465 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rwdwk","openshift-multus/multus-additional-cni-plugins-zxrbf","openshift-multus/multus-cmxnt","openshift-network-diagnostics/network-check-target-dm5ls","openshift-ovn-kubernetes/ovnkube-node-fzxzd","kube-system/global-pull-secret-syncer-g7gkl","kube-system/konnectivity-agent-p6j48","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h","openshift-dns/node-resolver-wd6zm","openshift-image-registry/node-ca-dc42q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal","openshift-multus/network-metrics-daemon-gg8ft","openshift-network-operator/iptables-alerter-w2jj8"] Sep 17 17:40:56.071271 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.071165 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.071271 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.071235 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:40:56.073410 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.073386 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:56.073513 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.073463 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:40:56.076149 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.075736 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.077941 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.077917 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.078435 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.078415 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 17:40:56.078564 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.078508 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 17:40:56.078637 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.078563 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dhfnq\"" Sep 17 17:40:56.078637 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.078574 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 17:40:56.078764 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.078508 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 17:40:56.078764 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.078660 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 17:40:56.080372 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.080177 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 17:40:56.080506 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.080489 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2fqsl\"" Sep 17 17:40:56.080681 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.080666 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 17:40:56.081453 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.080913 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:56.081453 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.080987 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:40:56.082739 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.082639 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.085967 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.085940 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 17:40:56.086076 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.085974 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 17:40:56.086194 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.086176 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 17:40:56.086322 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.086305 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 17:40:56.086389 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.086337 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 17:40:56.086447 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.086413 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 17:40:56.086679 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.086664 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t8zn5\"" Sep 17 17:40:56.087753 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.087314 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.089777 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.089753 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 17:40:56.089882 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.089785 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 17:40:56.089882 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.089756 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 17:40:56.090284 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.090266 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9j92\"" Sep 17 17:40:56.091580 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.091561 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.093028 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.092965 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:56.093123 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093027 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-cnibin\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.093123 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093054 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6847f4d4-2b04-47ca-82ee-399ef7517d45-cni-binary-copy\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093123 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093077 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2x59\" (UniqueName: \"kubernetes.io/projected/6847f4d4-2b04-47ca-82ee-399ef7517d45-kube-api-access-z2x59\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093123 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093107 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093131 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093155 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-system-cni-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093179 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2m9k\" (UniqueName: \"kubernetes.io/projected/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-kube-api-access-p2m9k\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093202 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-hostroot\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093225 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-etc-kubernetes\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093249 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-system-cni-dir\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093272 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-cni-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093304 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-socket-dir-parent\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093326 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-cni-multus\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093348 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-conf-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093370 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-multus-certs\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093429 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093471 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8cd2\" (UniqueName: \"kubernetes.io/projected/c4b5481b-65d4-49a4-827f-73254a600f11-kube-api-access-t8cd2\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093499 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-k8s-cni-cncf-io\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093551 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-daemon-config\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093577 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093600 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093624 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-cnibin\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093655 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-netns\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093677 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-cni-bin\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093703 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-kubelet\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.093742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093728 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.094503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093751 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-kubelet-config\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.094503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.093923 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 17:40:56.094503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.094017 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5sgb7\"" Sep 17 17:40:56.094503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.094193 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-dbus\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.094503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.094260 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.094503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.094285 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 17:40:56.094503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.094350 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 17:40:56.094842 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.094606 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 17:40:56.096878 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.096643 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-c5k52\"" Sep 17 17:40:56.096878 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.096657 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 17:40:56.096878 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.096746 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 17:40:56.098378 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.098104 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.100606 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.100588 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 17:40:56.100795 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.100773 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.100934 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.100599 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xqkf7\"" Sep 17 17:40:56.101252 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.101230 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 17:40:56.103024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.103003 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.103798 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.103380 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 17:40:56.103798 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.103427 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 17:40:56.103798 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.103484 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 17:40:56.103798 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.103380 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqwnt\"" Sep 17 17:40:56.105822 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.105802 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 17:40:56.105926 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.105870 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v22fw\"" Sep 17 17:40:56.105926 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.105873 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 17:40:56.124071 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.124035 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cpcrn" Sep 17 17:40:56.133393 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.133352 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cpcrn" Sep 17 17:40:56.179901 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.179866 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 17:40:56.195231 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195195 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-system-cni-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195237 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-modprobe-d\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195262 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-kubernetes\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195282 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-lib-modules\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195301 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-cni-bin\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195321 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-env-overrides\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195341 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovnkube-script-lib\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195366 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2m9k\" (UniqueName: \"kubernetes.io/projected/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-kube-api-access-p2m9k\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195389 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-hostroot\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.195420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195409 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysctl-d\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195422 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-system-cni-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195429 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970e8def-d389-4168-a5d2-17cb0cc42962-etc-tuned\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195490 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-systemd-units\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195519 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195640 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-hostroot\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195685 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovn-node-metrics-cert\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195722 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/173230ea-68b4-4625-9d03-bc0eaff8a310-host\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195751 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-cni-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195787 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-cni-multus\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195814 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-multus-certs\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195839 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195880 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lkmr\" (UniqueName: \"kubernetes.io/projected/3bcb8968-7f79-4c45-aab8-bc441842d62d-kube-api-access-9lkmr\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.195894 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195891 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-cni-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195904 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-run-netns\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195919 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-cni-multus\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195956 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.195977 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-multus-certs\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196026 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/416dd6dc-3054-408c-bbff-e1bc31acc67d-agent-certs\") pod \"konnectivity-agent-p6j48\" (UID: \"416dd6dc-3054-408c-bbff-e1bc31acc67d\") " pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196065 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8cd2\" (UniqueName: \"kubernetes.io/projected/c4b5481b-65d4-49a4-827f-73254a600f11-kube-api-access-t8cd2\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196100 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-k8s-cni-cncf-io\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196129 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/573ca411-6849-4ca8-885f-d3985df41a88-hosts-file\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196159 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-k8s-cni-cncf-io\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196183 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-cnibin\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196222 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-netns\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196237 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-cni-bin\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196264 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-cnibin\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196274 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-systemd\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-etc-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196304 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-run-netns\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.196478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196308 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-registration-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196342 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-cni-bin\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196374 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bcb8968-7f79-4c45-aab8-bc441842d62d-iptables-alerter-script\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196427 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970e8def-d389-4168-a5d2-17cb0cc42962-tmp\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196467 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-kubelet\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196511 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovnkube-config\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196558 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-kubelet\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196603 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196649 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-kubelet-config\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196687 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2x59\" (UniqueName: \"kubernetes.io/projected/6847f4d4-2b04-47ca-82ee-399ef7517d45-kube-api-access-z2x59\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196715 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-sys\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196738 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-kubelet-config\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196764 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-var-lib-kubelet\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196766 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-host-var-lib-kubelet\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196795 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196891 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-var-lib-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196930 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-log-socket\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.197411 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196960 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.196989 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197017 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lfk\" (UniqueName: \"kubernetes.io/projected/a18b81c5-0bad-4e81-b581-82fb42c3381d-kube-api-access-n4lfk\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197040 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjl9\" (UniqueName: \"kubernetes.io/projected/ddaaca48-ac39-4e85-8d85-f78b1467d084-kube-api-access-tpjl9\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197063 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-etc-kubernetes\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197086 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bcb8968-7f79-4c45-aab8-bc441842d62d-host-slash\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197106 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-host\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197129 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql5xm\" (UniqueName: \"kubernetes.io/projected/970e8def-d389-4168-a5d2-17cb0cc42962-kube-api-access-ql5xm\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197150 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-etc-kubernetes\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197152 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-systemd\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197191 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-node-log\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197215 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-system-cni-dir\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197239 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-socket-dir-parent\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197262 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-conf-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197276 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-system-cni-dir\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197285 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-device-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197319 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbm2l\" (UniqueName: \"kubernetes.io/projected/573ca411-6849-4ca8-885f-d3985df41a88-kube-api-access-jbm2l\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.198285 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197355 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-socket-dir-parent\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197363 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-conf-dir\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197385 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-slash\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197409 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-cni-netd\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197437 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197475 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-daemon-config\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197506 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-socket-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.197519 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197544 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysconfig\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197543 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197576 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysctl-conf\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.197603 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret podName:1eca5fda-05d1-43bf-8b60-a65f9480bcc3 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:56.69758216 +0000 UTC m=+2.998149683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret") pod "global-pull-secret-syncer-g7gkl" (UID: "1eca5fda-05d1-43bf-8b60-a65f9480bcc3") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197636 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197662 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197634 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197686 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.199060 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197739 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-sys-fs\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197764 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/173230ea-68b4-4625-9d03-bc0eaff8a310-serviceca\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197788 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdpz\" (UniqueName: \"kubernetes.io/projected/173230ea-68b4-4625-9d03-bc0eaff8a310-kube-api-access-lpdpz\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197809 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-run\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197833 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197865 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ca411-6849-4ca8-885f-d3985df41a88-tmp-dir\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197901 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-ovn\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197935 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-dbus\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.197985 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198036 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-cnibin\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198053 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-dbus\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198061 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6847f4d4-2b04-47ca-82ee-399ef7517d45-multus-daemon-config\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198062 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6847f4d4-2b04-47ca-82ee-399ef7517d45-cni-binary-copy\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.198095 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198101 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4b5481b-65d4-49a4-827f-73254a600f11-cnibin\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198106 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/416dd6dc-3054-408c-bbff-e1bc31acc67d-konnectivity-ca\") pod \"konnectivity-agent-p6j48\" (UID: \"416dd6dc-3054-408c-bbff-e1bc31acc67d\") " pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.198142 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs podName:a7996a81-8ee5-41db-a2a0-e3a1f6b13367 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:56.698127254 +0000 UTC m=+2.998694790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs") pod "network-metrics-daemon-gg8ft" (UID: "a7996a81-8ee5-41db-a2a0-e3a1f6b13367") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:56.199732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198107 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4b5481b-65d4-49a4-827f-73254a600f11-cni-binary-copy\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.200354 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.198486 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6847f4d4-2b04-47ca-82ee-399ef7517d45-cni-binary-copy\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.204396 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.204273 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:40:56.204396 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.204310 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:40:56.204396 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.204321 2550 projected.go:196] Error preparing data for projected volume kube-api-access-qlpzc for pod openshift-network-diagnostics/network-check-target-dm5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:40:56.204396 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.204375 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc podName:485f436b-3de3-4aeb-aec2-702f81e96c0a nodeName:}" failed. No retries permitted until 2026-09-17 17:40:56.70435953 +0000 UTC m=+3.004927064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qlpzc" (UniqueName: "kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc") pod "network-check-target-dm5ls" (UID: "485f436b-3de3-4aeb-aec2-702f81e96c0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:40:56.204674 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.204642 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 17:40:56.209078 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.209051 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2m9k\" (UniqueName: \"kubernetes.io/projected/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-kube-api-access-p2m9k\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:56.209078 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.209056 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2x59\" (UniqueName: \"kubernetes.io/projected/6847f4d4-2b04-47ca-82ee-399ef7517d45-kube-api-access-z2x59\") pod \"multus-cmxnt\" (UID: \"6847f4d4-2b04-47ca-82ee-399ef7517d45\") " pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.209269 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.209054 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8cd2\" (UniqueName: \"kubernetes.io/projected/c4b5481b-65d4-49a4-827f-73254a600f11-kube-api-access-t8cd2\") pod \"multus-additional-cni-plugins-zxrbf\" (UID: \"c4b5481b-65d4-49a4-827f-73254a600f11\") " pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.215372 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.215316 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" event={"ID":"323a5990fa2f22ac49cb118e6b347864","Type":"ContainerStarted","Data":"bd97465f31d95682e4d7f517b1c2f60a1b321c8b2a0822361862ca43e4cab0d2"} Sep 17 17:40:56.216714 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.216670 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" event={"ID":"bf772d8abc581fe5ccec09571c5d2950","Type":"ContainerStarted","Data":"5bae791eec3c7863e0a12c71e9f71bc9dfd075246d2f6f7b446a467e01ce58cf"} Sep 17 17:40:56.299220 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299184 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-cni-bin\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299220 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299226 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-env-overrides\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299249 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovnkube-script-lib\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299270 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-cni-bin\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299276 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysctl-d\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.299457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299321 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970e8def-d389-4168-a5d2-17cb0cc42962-etc-tuned\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.299457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299407 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysctl-d\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299470 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-systemd-units\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299509 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299555 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovn-node-metrics-cert\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299601 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-systemd-units\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299636 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/173230ea-68b4-4625-9d03-bc0eaff8a310-host\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299664 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299690 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lkmr\" (UniqueName: \"kubernetes.io/projected/3bcb8968-7f79-4c45-aab8-bc441842d62d-kube-api-access-9lkmr\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299712 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-run-netns\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.299735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299735 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299750 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299768 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/416dd6dc-3054-408c-bbff-e1bc31acc67d-agent-certs\") pod \"konnectivity-agent-p6j48\" (UID: \"416dd6dc-3054-408c-bbff-e1bc31acc67d\") " pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299794 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/573ca411-6849-4ca8-885f-d3985df41a88-hosts-file\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299821 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-systemd\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299875 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-etc-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299900 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-registration-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299923 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bcb8968-7f79-4c45-aab8-bc441842d62d-iptables-alerter-script\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299945 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970e8def-d389-4168-a5d2-17cb0cc42962-tmp\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299965 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-kubelet\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299990 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovnkube-config\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.299994 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-env-overrides\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300051 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovnkube-script-lib\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300020 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-sys\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300068 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-sys\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300030 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/173230ea-68b4-4625-9d03-bc0eaff8a310-host\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300077 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.300159 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300112 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-var-lib-kubelet\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300103 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300132 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-run-netns\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300181 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-var-lib-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-systemd\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300195 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-etc-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300207 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-log-socket\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300252 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-log-socket\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300286 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lfk\" (UniqueName: \"kubernetes.io/projected/a18b81c5-0bad-4e81-b581-82fb42c3381d-kube-api-access-n4lfk\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300326 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjl9\" (UniqueName: \"kubernetes.io/projected/ddaaca48-ac39-4e85-8d85-f78b1467d084-kube-api-access-tpjl9\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300351 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bcb8968-7f79-4c45-aab8-bc441842d62d-host-slash\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300373 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-host\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300421 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql5xm\" (UniqueName: \"kubernetes.io/projected/970e8def-d389-4168-a5d2-17cb0cc42962-kube-api-access-ql5xm\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300452 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-systemd\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300476 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-node-log\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300521 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-kubelet\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300672 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-registration-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.300969 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300426 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-host\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300715 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-var-lib-openvswitch\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300730 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3bcb8968-7f79-4c45-aab8-bc441842d62d-iptables-alerter-script\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300739 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-systemd\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300752 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovnkube-config\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300795 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/573ca411-6849-4ca8-885f-d3985df41a88-hosts-file\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300812 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-var-lib-kubelet\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300879 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-node-log\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300906 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-device-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300922 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3bcb8968-7f79-4c45-aab8-bc441842d62d-host-slash\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300974 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbm2l\" (UniqueName: \"kubernetes.io/projected/573ca411-6849-4ca8-885f-d3985df41a88-kube-api-access-jbm2l\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.300992 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-device-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301040 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-slash\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301066 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-slash\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301110 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-cni-netd\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301186 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-cni-netd\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301246 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-socket-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.301779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301283 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysconfig\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301326 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysctl-conf\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301391 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysconfig\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301392 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-socket-dir\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301436 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301462 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-sys-fs\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301485 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/173230ea-68b4-4625-9d03-bc0eaff8a310-serviceca\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301503 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-sysctl-conf\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301550 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdpz\" (UniqueName: \"kubernetes.io/projected/173230ea-68b4-4625-9d03-bc0eaff8a310-kube-api-access-lpdpz\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301581 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-run\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301605 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301660 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ca411-6849-4ca8-885f-d3985df41a88-tmp-dir\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301687 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-ovn\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301741 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-host-run-ovn-kubernetes\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301944 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-run\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301988 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-sys-fs\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301988 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a18b81c5-0bad-4e81-b581-82fb42c3381d-etc-selinux\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.302591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.301936 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/173230ea-68b4-4625-9d03-bc0eaff8a310-serviceca\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302133 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/416dd6dc-3054-408c-bbff-e1bc31acc67d-konnectivity-ca\") pod \"konnectivity-agent-p6j48\" (UID: \"416dd6dc-3054-408c-bbff-e1bc31acc67d\") " pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302174 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-modprobe-d\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302197 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-kubernetes\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302231 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-lib-modules\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302352 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-lib-modules\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302392 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ddaaca48-ac39-4e85-8d85-f78b1467d084-run-ovn\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302464 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-modprobe-d\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302507 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970e8def-d389-4168-a5d2-17cb0cc42962-etc-kubernetes\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302626 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/416dd6dc-3054-408c-bbff-e1bc31acc67d-agent-certs\") pod \"konnectivity-agent-p6j48\" (UID: \"416dd6dc-3054-408c-bbff-e1bc31acc67d\") " pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302713 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/416dd6dc-3054-408c-bbff-e1bc31acc67d-konnectivity-ca\") pod \"konnectivity-agent-p6j48\" (UID: \"416dd6dc-3054-408c-bbff-e1bc31acc67d\") " pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.303358 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.302987 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ddaaca48-ac39-4e85-8d85-f78b1467d084-ovn-node-metrics-cert\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.303888 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.303554 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/573ca411-6849-4ca8-885f-d3985df41a88-tmp-dir\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.304405 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.304383 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970e8def-d389-4168-a5d2-17cb0cc42962-tmp\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.305019 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.304986 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970e8def-d389-4168-a5d2-17cb0cc42962-etc-tuned\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.321543 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.321448 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lfk\" (UniqueName: \"kubernetes.io/projected/a18b81c5-0bad-4e81-b581-82fb42c3381d-kube-api-access-n4lfk\") pod \"aws-ebs-csi-driver-node-dct2h\" (UID: \"a18b81c5-0bad-4e81-b581-82fb42c3381d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.321543 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.321520 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjl9\" (UniqueName: \"kubernetes.io/projected/ddaaca48-ac39-4e85-8d85-f78b1467d084-kube-api-access-tpjl9\") pod \"ovnkube-node-fzxzd\" (UID: \"ddaaca48-ac39-4e85-8d85-f78b1467d084\") " pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.321788 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.321608 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdpz\" (UniqueName: \"kubernetes.io/projected/173230ea-68b4-4625-9d03-bc0eaff8a310-kube-api-access-lpdpz\") pod \"node-ca-dc42q\" (UID: \"173230ea-68b4-4625-9d03-bc0eaff8a310\") " pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.321788 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.321748 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbm2l\" (UniqueName: \"kubernetes.io/projected/573ca411-6849-4ca8-885f-d3985df41a88-kube-api-access-jbm2l\") pod \"node-resolver-wd6zm\" (UID: \"573ca411-6849-4ca8-885f-d3985df41a88\") " pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.323169 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.323147 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql5xm\" (UniqueName: \"kubernetes.io/projected/970e8def-d389-4168-a5d2-17cb0cc42962-kube-api-access-ql5xm\") pod \"tuned-rwdwk\" (UID: \"970e8def-d389-4168-a5d2-17cb0cc42962\") " pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.323466 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.323424 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lkmr\" (UniqueName: \"kubernetes.io/projected/3bcb8968-7f79-4c45-aab8-bc441842d62d-kube-api-access-9lkmr\") pod \"iptables-alerter-w2jj8\" (UID: \"3bcb8968-7f79-4c45-aab8-bc441842d62d\") " pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.391595 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.391560 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" Sep 17 17:40:56.400504 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.400475 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cmxnt" Sep 17 17:40:56.409614 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.409566 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:40:56.415679 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.415643 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dc42q" Sep 17 17:40:56.425846 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.425813 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" Sep 17 17:40:56.434926 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.434901 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w2jj8" Sep 17 17:40:56.443131 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.443102 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" Sep 17 17:40:56.451001 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.450971 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd6zm" Sep 17 17:40:56.456020 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.455998 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:40:56.705661 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.705540 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:56.705661 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.705620 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:56.705661 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:56.705657 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705730 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705757 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705773 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705781 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705824 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret podName:1eca5fda-05d1-43bf-8b60-a65f9480bcc3 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:57.705808706 +0000 UTC m=+4.006376233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret") pod "global-pull-secret-syncer-g7gkl" (UID: "1eca5fda-05d1-43bf-8b60-a65f9480bcc3") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705830 2550 projected.go:196] Error preparing data for projected volume kube-api-access-qlpzc for pod openshift-network-diagnostics/network-check-target-dm5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705844 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs podName:a7996a81-8ee5-41db-a2a0-e3a1f6b13367 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:57.705835454 +0000 UTC m=+4.006402977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs") pod "network-metrics-daemon-gg8ft" (UID: "a7996a81-8ee5-41db-a2a0-e3a1f6b13367") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:56.705953 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:56.705869 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc podName:485f436b-3de3-4aeb-aec2-702f81e96c0a nodeName:}" failed. No retries permitted until 2026-09-17 17:40:57.705853581 +0000 UTC m=+4.006421124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlpzc" (UniqueName: "kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc") pod "network-check-target-dm5ls" (UID: "485f436b-3de3-4aeb-aec2-702f81e96c0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:40:57.137483 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:57.137341 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:35:56 +0000 UTC" deadline="2028-04-18 16:41:51.077246786 +0000 UTC" Sep 17 17:40:57.137483 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:57.137421 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13895h0m53.939830989s" Sep 17 17:40:57.209630 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:57.209583 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:57.209798 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.209744 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:40:57.332990 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:57.332779 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573ca411_6849_4ca8_885f_d3985df41a88.slice/crio-0a57e0b1f5ae026331bd76a2da9aabb6a12a19739b1fc2fe6915168424ec7743 WatchSource:0}: Error finding container 0a57e0b1f5ae026331bd76a2da9aabb6a12a19739b1fc2fe6915168424ec7743: Status 404 returned error can't find the container with id 0a57e0b1f5ae026331bd76a2da9aabb6a12a19739b1fc2fe6915168424ec7743 Sep 17 17:40:57.335534 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:57.335489 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18b81c5_0bad_4e81_b581_82fb42c3381d.slice/crio-59b2a431ad2ab3a47bc58dbe17beb67cf22ccdcec796a19174d43ab72f131336 WatchSource:0}: Error finding container 59b2a431ad2ab3a47bc58dbe17beb67cf22ccdcec796a19174d43ab72f131336: Status 404 returned error can't find the container with id 59b2a431ad2ab3a47bc58dbe17beb67cf22ccdcec796a19174d43ab72f131336 Sep 17 17:40:57.339014 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:57.338989 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416dd6dc_3054_408c_bbff_e1bc31acc67d.slice/crio-a786ebd79a500d3f21fa10bc35b8d0935bf4b0566441640284f63459cfdd9372 WatchSource:0}: Error finding container a786ebd79a500d3f21fa10bc35b8d0935bf4b0566441640284f63459cfdd9372: Status 404 returned error can't find the container with id a786ebd79a500d3f21fa10bc35b8d0935bf4b0566441640284f63459cfdd9372 Sep 17 17:40:57.341171 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:40:57.341150 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6847f4d4_2b04_47ca_82ee_399ef7517d45.slice/crio-88977d9f106c17c2134b85394f93d222170904224f7aa8fcba9971fccc378f3e WatchSource:0}: Error finding container 88977d9f106c17c2134b85394f93d222170904224f7aa8fcba9971fccc378f3e: Status 404 returned error can't find the container with id 88977d9f106c17c2134b85394f93d222170904224f7aa8fcba9971fccc378f3e Sep 17 17:40:57.713654 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:57.713567 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:57.713654 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:57.713614 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:57.713654 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:57.713648 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:57.713873 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713698 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:57.713873 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713764 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret podName:1eca5fda-05d1-43bf-8b60-a65f9480bcc3 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.71374346 +0000 UTC m=+6.014310995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret") pod "global-pull-secret-syncer-g7gkl" (UID: "1eca5fda-05d1-43bf-8b60-a65f9480bcc3") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:57.713873 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713772 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:57.713873 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713808 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs podName:a7996a81-8ee5-41db-a2a0-e3a1f6b13367 nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.713795839 +0000 UTC m=+6.014363377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs") pod "network-metrics-daemon-gg8ft" (UID: "a7996a81-8ee5-41db-a2a0-e3a1f6b13367") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:57.714032 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713885 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:40:57.714032 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713899 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:40:57.714032 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713912 2550 projected.go:196] Error preparing data for projected volume kube-api-access-qlpzc for pod openshift-network-diagnostics/network-check-target-dm5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:40:57.714032 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:57.713945 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc podName:485f436b-3de3-4aeb-aec2-702f81e96c0a nodeName:}" failed. No retries permitted until 2026-09-17 17:40:59.713933252 +0000 UTC m=+6.014500775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlpzc" (UniqueName: "kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc") pod "network-check-target-dm5ls" (UID: "485f436b-3de3-4aeb-aec2-702f81e96c0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:40:58.138774 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.138667 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:35:56 +0000 UTC" deadline="2028-05-08 12:57:14.666105232 +0000 UTC" Sep 17 17:40:58.138774 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.138715 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14371h16m16.527396218s" Sep 17 17:40:58.212354 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.212315 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:58.212559 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:58.212470 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:40:58.212922 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.212900 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:58.213741 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:58.213021 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:40:58.238709 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.238658 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6j48" event={"ID":"416dd6dc-3054-408c-bbff-e1bc31acc67d","Type":"ContainerStarted","Data":"a786ebd79a500d3f21fa10bc35b8d0935bf4b0566441640284f63459cfdd9372"} Sep 17 17:40:58.243038 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.243000 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" event={"ID":"a18b81c5-0bad-4e81-b581-82fb42c3381d","Type":"ContainerStarted","Data":"59b2a431ad2ab3a47bc58dbe17beb67cf22ccdcec796a19174d43ab72f131336"} Sep 17 17:40:58.247672 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.247638 2550 generic.go:356] "Generic (PLEG): container finished" podID="bf772d8abc581fe5ccec09571c5d2950" containerID="e0cf2ae180dea4d8f39fe4239cdefbb559c0a82578726b78764afd5967666ca1" exitCode=0 Sep 17 17:40:58.247923 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.247891 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" event={"ID":"bf772d8abc581fe5ccec09571c5d2950","Type":"ContainerDied","Data":"e0cf2ae180dea4d8f39fe4239cdefbb559c0a82578726b78764afd5967666ca1"} Sep 17 17:40:58.254098 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.254060 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dc42q" event={"ID":"173230ea-68b4-4625-9d03-bc0eaff8a310","Type":"ContainerStarted","Data":"1fa66db320514d4acd8e588387c8f93d7b43542010526b38b9e7fec109d640eb"} Sep 17 17:40:58.265624 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.265583 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" event={"ID":"970e8def-d389-4168-a5d2-17cb0cc42962","Type":"ContainerStarted","Data":"06ab5e33399d4a006f00e9e8557cec45729f2431c3e84ee435d03c69fc247235"} Sep 17 17:40:58.274522 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.274444 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cmxnt" event={"ID":"6847f4d4-2b04-47ca-82ee-399ef7517d45","Type":"ContainerStarted","Data":"88977d9f106c17c2134b85394f93d222170904224f7aa8fcba9971fccc378f3e"} Sep 17 17:40:58.276271 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.276189 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd6zm" event={"ID":"573ca411-6849-4ca8-885f-d3985df41a88","Type":"ContainerStarted","Data":"0a57e0b1f5ae026331bd76a2da9aabb6a12a19739b1fc2fe6915168424ec7743"} Sep 17 17:40:58.288179 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.288139 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" event={"ID":"323a5990fa2f22ac49cb118e6b347864","Type":"ContainerStarted","Data":"2f8638a21d55a70e96cf70f7105f84f0494fdeee9c2e425a4797f04a248f1765"} Sep 17 17:40:58.303920 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.303877 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerStarted","Data":"e152123a0bceaf769f75d8152cb81bad0ef4e433143b816695554a95c89d44ea"} Sep 17 17:40:58.316866 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.316727 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"afe8d032947139fe111261ec74bab095fd2eb9b136f66e3ae4733deace2c2098"} Sep 17 17:40:58.333785 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:58.333726 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w2jj8" event={"ID":"3bcb8968-7f79-4c45-aab8-bc441842d62d","Type":"ContainerStarted","Data":"50c6fb98de6547ffea61105c1ff3a4985e04ee19e6b989bb2e449fbb044205b2"} Sep 17 17:40:59.210211 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:59.210017 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:59.210902 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.210213 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:40:59.366844 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:59.366773 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" event={"ID":"bf772d8abc581fe5ccec09571c5d2950","Type":"ContainerStarted","Data":"a3a43bf4903299f72fdee0208e7f116ee7f5c8da92c8450708c95c94ada7f69e"} Sep 17 17:40:59.390847 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:59.390768 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-40.ec2.internal" podStartSLOduration=3.3907415 podStartE2EDuration="3.3907415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:58.350289883 +0000 UTC m=+4.650857431" watchObservedRunningTime="2026-09-17 17:40:59.3907415 +0000 UTC m=+5.691309048" Sep 17 17:40:59.391905 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:59.391829 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-40.ec2.internal" podStartSLOduration=3.391817202 podStartE2EDuration="3.391817202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:40:59.391401266 +0000 UTC m=+5.691968839" watchObservedRunningTime="2026-09-17 17:40:59.391817202 +0000 UTC m=+5.692384750" Sep 17 17:40:59.734638 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:59.734586 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:40:59.734638 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:59.734636 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:40:59.734875 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:40:59.734669 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:40:59.734875 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.734794 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:59.734875 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.734859 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs podName:a7996a81-8ee5-41db-a2a0-e3a1f6b13367 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:03.734839803 +0000 UTC m=+10.035407347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs") pod "network-metrics-daemon-gg8ft" (UID: "a7996a81-8ee5-41db-a2a0-e3a1f6b13367") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:40:59.735310 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.735269 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:59.735410 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.735327 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret podName:1eca5fda-05d1-43bf-8b60-a65f9480bcc3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:03.735310693 +0000 UTC m=+10.035878217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret") pod "global-pull-secret-syncer-g7gkl" (UID: "1eca5fda-05d1-43bf-8b60-a65f9480bcc3") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:40:59.735481 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.735425 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:40:59.735481 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.735444 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:40:59.735481 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.735458 2550 projected.go:196] Error preparing data for projected volume kube-api-access-qlpzc for pod openshift-network-diagnostics/network-check-target-dm5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:40:59.735657 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:40:59.735501 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc podName:485f436b-3de3-4aeb-aec2-702f81e96c0a nodeName:}" failed. No retries permitted until 2026-09-17 17:41:03.735486806 +0000 UTC m=+10.036054345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlpzc" (UniqueName: "kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc") pod "network-check-target-dm5ls" (UID: "485f436b-3de3-4aeb-aec2-702f81e96c0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:41:00.212705 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:00.212536 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:00.213261 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:00.212722 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:00.213261 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:00.213157 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:00.213408 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:00.213274 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:01.210324 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:01.210288 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:01.210533 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:01.210426 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:02.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:02.210570 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:02.211138 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:02.210695 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:02.211138 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:02.211096 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:02.211285 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:02.211206 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:03.211336 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:03.210797 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:03.211336 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.210957 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:03.772004 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:03.772061 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:03.772096 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772230 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772297 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs podName:a7996a81-8ee5-41db-a2a0-e3a1f6b13367 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.772276499 +0000 UTC m=+18.072844028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs") pod "network-metrics-daemon-gg8ft" (UID: "a7996a81-8ee5-41db-a2a0-e3a1f6b13367") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772594 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772635 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772649 2550 projected.go:196] Error preparing data for projected volume kube-api-access-qlpzc for pod openshift-network-diagnostics/network-check-target-dm5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772699 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc podName:485f436b-3de3-4aeb-aec2-702f81e96c0a nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.772683184 +0000 UTC m=+18.073250729 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlpzc" (UniqueName: "kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc") pod "network-check-target-dm5ls" (UID: "485f436b-3de3-4aeb-aec2-702f81e96c0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772813 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:41:03.772888 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:03.772863 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret podName:1eca5fda-05d1-43bf-8b60-a65f9480bcc3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:11.772846912 +0000 UTC m=+18.073414458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret") pod "global-pull-secret-syncer-g7gkl" (UID: "1eca5fda-05d1-43bf-8b60-a65f9480bcc3") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:41:04.211398 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:04.211312 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:04.211920 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:04.211438 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:04.211920 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:04.211647 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:04.211920 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:04.211738 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:05.210269 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:05.210236 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:05.210476 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:05.210352 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:06.211251 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:06.211144 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:06.211251 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:06.211209 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:06.211832 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:06.211320 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:06.211832 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:06.211469 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:07.209583 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:07.209523 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:07.209786 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:07.209687 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:08.210312 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:08.210020 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:08.210312 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:08.210067 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:08.210312 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:08.210182 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:08.210312 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:08.210268 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:09.209848 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:09.209811 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:09.210031 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:09.209921 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:10.210119 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:10.209918 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:10.210601 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:10.209918 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:10.210601 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:10.210231 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:10.210601 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:10.210260 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:11.210140 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:11.210095 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:11.210626 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.210218 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:11.837170 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:11.837125 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:11.837366 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:11.837184 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:11.837366 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:11.837221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:11.837366 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837273 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:41:11.837366 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837337 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:41:11.837366 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837353 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret podName:1eca5fda-05d1-43bf-8b60-a65f9480bcc3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.83733339 +0000 UTC m=+34.137900925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret") pod "global-pull-secret-syncer-g7gkl" (UID: "1eca5fda-05d1-43bf-8b60-a65f9480bcc3") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:41:11.837643 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837364 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:41:11.837643 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837387 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs podName:a7996a81-8ee5-41db-a2a0-e3a1f6b13367 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.837370474 +0000 UTC m=+34.137938015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs") pod "network-metrics-daemon-gg8ft" (UID: "a7996a81-8ee5-41db-a2a0-e3a1f6b13367") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:41:11.837643 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837387 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:41:11.837643 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837406 2550 projected.go:196] Error preparing data for projected volume kube-api-access-qlpzc for pod openshift-network-diagnostics/network-check-target-dm5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:41:11.837643 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:11.837430 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc podName:485f436b-3de3-4aeb-aec2-702f81e96c0a nodeName:}" failed. No retries permitted until 2026-09-17 17:41:27.8374235 +0000 UTC m=+34.137991021 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlpzc" (UniqueName: "kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc") pod "network-check-target-dm5ls" (UID: "485f436b-3de3-4aeb-aec2-702f81e96c0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:41:12.210191 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:12.210160 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:12.210549 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:12.210200 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:12.210549 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:12.210297 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:12.210549 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:12.210394 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:13.209494 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:13.209459 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:13.209681 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:13.209602 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:14.211149 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:14.211101 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:14.211591 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:14.211241 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:14.211591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:14.211304 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:14.211591 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:14.211404 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:15.209543 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:15.209487 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:15.209734 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:15.209634 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:16.210431 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.210397 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:16.211010 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.210408 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:16.211010 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:16.210565 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:16.211010 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:16.210610 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:16.403862 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.403832 2550 generic.go:356] "Generic (PLEG): container finished" podID="c4b5481b-65d4-49a4-827f-73254a600f11" containerID="311f32ad6d325055d6a91cf6d7270f9286653b00c19051c429317e7aa98ca1b5" exitCode=0 Sep 17 17:41:16.404054 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.403923 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerDied","Data":"311f32ad6d325055d6a91cf6d7270f9286653b00c19051c429317e7aa98ca1b5"} Sep 17 17:41:16.406297 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.406267 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"abda8d25d34e87da83889ec9c807db00e498476b2096a48536171d4a3b442d58"} Sep 17 17:41:16.406389 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.406363 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"ea6eeda2f9b1aec1932247211094bddd8ec58665048ac5f6891ea051152901ef"} Sep 17 17:41:16.406465 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.406410 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"d4d2fa850a1d1bccc44112df3e53a2e589a396b8c8d2b61e1d233d44e61e1e9c"} Sep 17 17:41:16.407712 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.407690 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-p6j48" event={"ID":"416dd6dc-3054-408c-bbff-e1bc31acc67d","Type":"ContainerStarted","Data":"0c5c8b5565602f3a24a2c9b970e01740344b7fe8a716050c03b91cb7c4e861f3"} Sep 17 17:41:16.409288 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.409265 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" event={"ID":"a18b81c5-0bad-4e81-b581-82fb42c3381d","Type":"ContainerStarted","Data":"08387eda1295c4d51d4750a2c0aecefb35a873862cff3dd6f326c7242aac42fa"} Sep 17 17:41:16.410716 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.410695 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dc42q" event={"ID":"173230ea-68b4-4625-9d03-bc0eaff8a310","Type":"ContainerStarted","Data":"7cdd1abf94958187ba51ac97b702f0aa0bfa365c89cc8bbe4019ad13edec836e"} Sep 17 17:41:16.412307 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.412286 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" event={"ID":"970e8def-d389-4168-a5d2-17cb0cc42962","Type":"ContainerStarted","Data":"ddd839aec482a46abbec9a1ca5d0cad67c81a622c858e0a6e7016d4b501acf78"} Sep 17 17:41:16.413559 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.413516 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cmxnt" event={"ID":"6847f4d4-2b04-47ca-82ee-399ef7517d45","Type":"ContainerStarted","Data":"f91ebcfde5a1bec3fb5d339906809b4e908b94196d2d21b09c87d9d81ac64fa1"} Sep 17 17:41:16.414748 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.414729 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd6zm" event={"ID":"573ca411-6849-4ca8-885f-d3985df41a88","Type":"ContainerStarted","Data":"34d5aada3541e98b19c20ec4eb44c41c99dbf8c11286b971abe17ee3e509f993"} Sep 17 17:41:16.443651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.443595 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rwdwk" podStartSLOduration=2.928145841 podStartE2EDuration="21.443576076s" totalImagesPullingTime="18.515430235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:16.443334754 +0000 UTC m=+22.743902300" watchObservedRunningTime="2026-09-17 17:41:16.443576076 +0000 UTC m=+22.744143665" Sep 17 17:41:16.458443 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.458399 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wd6zm" podStartSLOduration=2.957868645 podStartE2EDuration="21.458384696s" totalImagesPullingTime="18.500516051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:16.45806372 +0000 UTC m=+22.758631263" watchObservedRunningTime="2026-09-17 17:41:16.458384696 +0000 UTC m=+22.758952240" Sep 17 17:41:16.476233 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.476177 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dc42q" podStartSLOduration=2.997657787 podStartE2EDuration="21.476161237s" totalImagesPullingTime="18.47850345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:16.475684911 +0000 UTC m=+22.776252455" watchObservedRunningTime="2026-09-17 17:41:16.476161237 +0000 UTC m=+22.776728782" Sep 17 17:41:16.512323 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.512276 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cmxnt" podStartSLOduration=2.970587815 podStartE2EDuration="21.512260726s" totalImagesPullingTime="18.541672911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:16.511768319 +0000 UTC m=+22.812335862" watchObservedRunningTime="2026-09-17 17:41:16.512260726 +0000 UTC m=+22.812828269" Sep 17 17:41:16.512586 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.512553 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-p6j48" podStartSLOduration=3.024045341 podStartE2EDuration="21.512545676s" totalImagesPullingTime="18.488500335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:16.492567921 +0000 UTC m=+22.793135459" watchObservedRunningTime="2026-09-17 17:41:16.512545676 +0000 UTC m=+22.813113218" Sep 17 17:41:16.653609 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.653419 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:16.714836 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:16.714797 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:17.134053 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.134013 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:17.210142 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.210106 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:17.210333 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:17.210228 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:17.233702 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.233660 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:17.390719 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.390649 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 17:41:17.419009 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.418965 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w2jj8" event={"ID":"3bcb8968-7f79-4c45-aab8-bc441842d62d","Type":"ContainerStarted","Data":"b365f181d432625026120a84b74e1f56dc16b41c05a341cdf335b552dc14d120"} Sep 17 17:41:17.421135 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.421105 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" event={"ID":"a18b81c5-0bad-4e81-b581-82fb42c3381d","Type":"ContainerStarted","Data":"bf32a0f1ced07c92ea89d63a56239080a2d6c468f8798a2d03f853fa003405ff"} Sep 17 17:41:17.424592 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.424553 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"c5c71de1c262259c867ac69ba48d219e441d42aeb3267ac56515f34b1078fb2d"} Sep 17 17:41:17.424725 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.424633 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"2611de839cc33e2cee2247e1bc7cdbecb8160d7168788ce14e91e84f3bd60adc"} Sep 17 17:41:17.424725 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.424681 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"0e7b3f518dcb771267f168acd882adde8767e2e50194fa3134dad2208b118636"} Sep 17 17:41:17.555238 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.555204 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:17.656758 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:17.656658 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:18.149975 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:18.149845 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T17:41:17.39066939Z","UUID":"79f990ef-695f-451a-bf0b-2680141e809b","Handler":null,"Name":"","Endpoint":""} Sep 17 17:41:18.151547 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:18.151515 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 17:41:18.151678 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:18.151566 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 17:41:18.209943 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:18.209869 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:18.210176 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:18.210008 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:18.210455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:18.210395 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:18.210595 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:18.210520 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:19.210110 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:19.210072 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:19.210548 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:19.210203 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:19.433315 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:19.433265 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"4160c060cda092ef9a4e4ff88b382f8da3bd5bd9fc4e6cfc52ab548db4bc5a7a"} Sep 17 17:41:19.435578 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:19.435542 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" event={"ID":"a18b81c5-0bad-4e81-b581-82fb42c3381d","Type":"ContainerStarted","Data":"b1add4cc2fbc0c6ba86023cfeefef0ff756579be014029d99c536fcd39d9b2e8"} Sep 17 17:41:19.459861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:19.459809 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w2jj8" podStartSLOduration=5.962328395 podStartE2EDuration="24.45979305s" totalImagesPullingTime="18.497464655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:17.434780585 +0000 UTC m=+23.735348130" watchObservedRunningTime="2026-09-17 17:41:19.45979305 +0000 UTC m=+25.760360594" Sep 17 17:41:19.460455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:19.460369 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dct2h" podStartSLOduration=3.7832700360000002 podStartE2EDuration="24.460356429s" totalImagesPullingTime="20.677086393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:19.45964843 +0000 UTC m=+25.760215975" watchObservedRunningTime="2026-09-17 17:41:19.460356429 +0000 UTC m=+25.760923972" Sep 17 17:41:20.177319 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:20.177281 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:20.210362 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:20.210326 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:20.210822 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:20.210328 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:20.210822 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:20.210469 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:20.210822 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:20.210593 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:20.251451 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:20.251415 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:20.588015 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:20.587924 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:41:20.588779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:20.588756 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:41:20.988734 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:20.988705 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:21.082235 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.081524 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:21.210279 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.210120 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:21.210374 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:21.210356 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:21.221928 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.221892 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:21.385859 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.385823 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:21.442029 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.441993 2550 generic.go:356] "Generic (PLEG): container finished" podID="c4b5481b-65d4-49a4-827f-73254a600f11" containerID="2958d1ba00b098486281932d10c13fe95259459bf3d7564c8e94748f33761b2f" exitCode=0 Sep 17 17:41:21.442177 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.442077 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerDied","Data":"2958d1ba00b098486281932d10c13fe95259459bf3d7564c8e94748f33761b2f"} Sep 17 17:41:21.446116 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.446089 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" event={"ID":"ddaaca48-ac39-4e85-8d85-f78b1467d084","Type":"ContainerStarted","Data":"e97985cfc148aa157a22c4c922e65e46fb3e2008e6463db293029b1eaf4a7322"} Sep 17 17:41:21.446395 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.446371 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:41:21.446788 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.446768 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-p6j48" Sep 17 17:41:21.468543 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.468494 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:21.508137 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.508086 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" podStartSLOduration=7.940167913 podStartE2EDuration="26.508067472s" totalImagesPullingTime="18.567899559s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:21.50804021 +0000 UTC m=+27.808607755" watchObservedRunningTime="2026-09-17 17:41:21.508067472 +0000 UTC m=+27.808634997" Sep 17 17:41:21.540806 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.540771 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:21.730992 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.730953 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:21.824469 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:21.824420 2550 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" limit=1024 Sep 17 17:41:22.210462 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.210415 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:22.210650 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:22.210545 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:22.210650 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.210580 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:22.210729 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:22.210668 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:22.449205 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.448983 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:41:22.449205 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.449075 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:41:22.449205 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.449126 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:41:22.464302 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.464218 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:41:22.464432 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.464320 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:41:22.578426 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:22.578400 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd6zm_573ca411-6849-4ca8-885f-d3985df41a88/dns-node-resolver/0.log" Sep 17 17:41:23.175912 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.175877 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7gkl"] Sep 17 17:41:23.176075 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.176049 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:23.176157 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:23.176136 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:23.178769 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.178743 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gg8ft"] Sep 17 17:41:23.178906 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.178853 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:23.178987 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:23.178966 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:23.179398 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.179376 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dm5ls"] Sep 17 17:41:23.179475 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.179465 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:23.179587 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:23.179567 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:23.376378 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.376304 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dc42q_173230ea-68b4-4625-9d03-bc0eaff8a310/node-ca/0.log" Sep 17 17:41:23.452455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.452418 2550 generic.go:356] "Generic (PLEG): container finished" podID="c4b5481b-65d4-49a4-827f-73254a600f11" containerID="41e40deb0cf9bd15e50c48b25f6d1cc35904797dfadb505cdf24dff425eee08e" exitCode=0 Sep 17 17:41:23.453156 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:23.452490 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerDied","Data":"41e40deb0cf9bd15e50c48b25f6d1cc35904797dfadb505cdf24dff425eee08e"} Sep 17 17:41:24.457138 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:24.456880 2550 generic.go:356] "Generic (PLEG): container finished" podID="c4b5481b-65d4-49a4-827f-73254a600f11" containerID="2a307be6a9670fc15cc4336b3de16535dafe44d3b2179391dd55f747f9dc5b9d" exitCode=0 Sep 17 17:41:24.457559 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:24.456949 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerDied","Data":"2a307be6a9670fc15cc4336b3de16535dafe44d3b2179391dd55f747f9dc5b9d"} Sep 17 17:41:25.210350 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:25.210315 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:25.210520 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:25.210418 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:25.210520 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:25.210430 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:25.210520 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:25.210425 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:25.210718 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:25.210522 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:25.210718 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:25.210632 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:27.210335 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:27.210300 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:27.210335 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:27.210337 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:27.210904 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:27.210342 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:27.210904 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.210423 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:27.210904 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.210505 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:27.210904 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.210616 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:27.862219 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:27.862181 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:27.862232 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:27.862263 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862354 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862375 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862409 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862423 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret podName:1eca5fda-05d1-43bf-8b60-a65f9480bcc3 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:59.862406091 +0000 UTC m=+66.162973633 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret") pod "global-pull-secret-syncer-g7gkl" (UID: "1eca5fda-05d1-43bf-8b60-a65f9480bcc3") : object "kube-system"/"original-pull-secret" not registered Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862425 2550 projected.go:196] Error preparing data for projected volume kube-api-access-qlpzc for pod openshift-network-diagnostics/network-check-target-dm5ls: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862465 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc podName:485f436b-3de3-4aeb-aec2-702f81e96c0a nodeName:}" failed. No retries permitted until 2026-09-17 17:41:59.86245645 +0000 UTC m=+66.163023972 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qlpzc" (UniqueName: "kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc") pod "network-check-target-dm5ls" (UID: "485f436b-3de3-4aeb-aec2-702f81e96c0a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862381 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:41:27.862558 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:27.862496 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs podName:a7996a81-8ee5-41db-a2a0-e3a1f6b13367 nodeName:}" failed. No retries permitted until 2026-09-17 17:41:59.862489299 +0000 UTC m=+66.163056853 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs") pod "network-metrics-daemon-gg8ft" (UID: "a7996a81-8ee5-41db-a2a0-e3a1f6b13367") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 17:41:29.210311 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:29.210276 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:29.210721 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:29.210372 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:29.210721 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:29.210276 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:29.210721 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:29.210276 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:29.210721 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:29.210440 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:29.210721 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:29.210566 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:29.472207 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:29.472174 2550 generic.go:356] "Generic (PLEG): container finished" podID="c4b5481b-65d4-49a4-827f-73254a600f11" containerID="93928e241596d1f601e630757734324892e12925e9a4fd18a932acd26168d53d" exitCode=0 Sep 17 17:41:29.472443 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:29.472238 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerDied","Data":"93928e241596d1f601e630757734324892e12925e9a4fd18a932acd26168d53d"} Sep 17 17:41:30.477416 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:30.477383 2550 generic.go:356] "Generic (PLEG): container finished" podID="c4b5481b-65d4-49a4-827f-73254a600f11" containerID="1d9f836e061e5bf2a98ce2bb297c3e8cb55d87078b10427013802e65038c7c74" exitCode=0 Sep 17 17:41:30.477922 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:30.477452 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerDied","Data":"1d9f836e061e5bf2a98ce2bb297c3e8cb55d87078b10427013802e65038c7c74"} Sep 17 17:41:31.210037 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:31.209998 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:31.210240 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:31.210102 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:31.210240 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:31.210109 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:31.210240 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:31.210208 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:31.210240 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:31.210111 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:31.210454 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:31.210321 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:31.483896 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:31.483803 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" event={"ID":"c4b5481b-65d4-49a4-827f-73254a600f11","Type":"ContainerStarted","Data":"94b226346a661c6f4342386b936c5feaa23a0897b5d8de92ed4ae2bd3346f207"} Sep 17 17:41:31.511236 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:31.511180 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zxrbf" podStartSLOduration=6.947983783 podStartE2EDuration="36.51116212s" totalImagesPullingTime="29.563178337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:31.510173076 +0000 UTC m=+37.810740634" watchObservedRunningTime="2026-09-17 17:41:31.51116212 +0000 UTC m=+37.811729658" Sep 17 17:41:33.210257 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:33.210216 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:33.210257 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:33.210236 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:33.210793 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:33.210323 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:33.210793 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:33.210324 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:33.210793 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:33.210401 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:33.210793 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:33.210475 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:35.210242 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:35.210204 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:35.210718 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:35.210305 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:35.210718 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:35.210313 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:35.210718 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:35.210332 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:35.210718 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:35.210402 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:35.210718 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:35.210479 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:37.209550 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:37.209502 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:37.209972 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:37.209502 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:37.209972 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:37.209638 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:37.209972 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:37.209517 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:37.209972 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:37.209710 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:37.209972 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:37.209801 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:39.209855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:39.209818 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:39.210260 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:39.209920 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:39.210260 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:39.209928 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:39.210260 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:39.209946 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:39.210260 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:39.210017 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:39.210260 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:39.210083 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:41.209823 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:41.209789 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:41.210177 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:41.209839 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:41.210177 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:41.209805 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:41.210177 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:41.209905 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:41.210177 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:41.209954 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:41.210177 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:41.210041 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:43.209963 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:43.209935 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:43.210337 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:43.210020 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:43.210337 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:43.210054 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:43.210337 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:43.210114 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g7gkl" podUID="1eca5fda-05d1-43bf-8b60-a65f9480bcc3" Sep 17 17:41:43.210337 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:43.210023 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dm5ls" podUID="485f436b-3de3-4aeb-aec2-702f81e96c0a" Sep 17 17:41:43.210337 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:41:43.210203 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gg8ft" podUID="a7996a81-8ee5-41db-a2a0-e3a1f6b13367" Sep 17 17:41:44.927850 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.927820 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-40.ec2.internal" event="NodeReady" Sep 17 17:41:44.928233 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.927941 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 17:41:44.964328 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.964290 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bc47694cb-555dv"] Sep 17 17:41:44.974478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.974450 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:44.977956 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.977930 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 17:41:44.978096 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.978020 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 17:41:44.978096 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.978075 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 17:41:44.978210 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.978179 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z4x8c\"" Sep 17 17:41:44.981868 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.981792 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc47694cb-555dv"] Sep 17 17:41:44.982519 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.982484 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6f52j"] Sep 17 17:41:44.986142 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.986117 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 17:41:44.999557 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.999509 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nb9lq"] Sep 17 17:41:44.999788 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:44.999678 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.003084 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.003061 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gsgx7\"" Sep 17 17:41:45.003470 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.003437 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 17:41:45.003615 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.003548 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 17:41:45.011258 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.011239 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6f52j"] Sep 17 17:41:45.011357 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.011270 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nb9lq"] Sep 17 17:41:45.011357 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.011344 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.014387 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.014366 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 17:41:45.014580 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.014555 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tzqnk\"" Sep 17 17:41:45.014711 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.014695 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 17:41:45.014866 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.014846 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 17:41:45.077363 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077318 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmq7\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-kube-api-access-jpmq7\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.077363 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077368 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54af9832-412c-4a7d-9503-a7f84b6aafad-ca-trust-extracted\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.077620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077389 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-registry-tls\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.077620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077445 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54af9832-412c-4a7d-9503-a7f84b6aafad-registry-certificates\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.077620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077464 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-bound-sa-token\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.077620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077514 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54af9832-412c-4a7d-9503-a7f84b6aafad-image-registry-private-configuration\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.077620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077586 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54af9832-412c-4a7d-9503-a7f84b6aafad-trusted-ca\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.077620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.077602 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54af9832-412c-4a7d-9503-a7f84b6aafad-installation-pull-secrets\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.083316 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.083288 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zrbjn"] Sep 17 17:41:45.099173 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.099145 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zrbjn"] Sep 17 17:41:45.099305 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.099267 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.101954 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.101923 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 17:41:45.102077 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.101954 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 17:41:45.102077 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.101998 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 17:41:45.102077 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.102063 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 17:41:45.102266 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.102104 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-b7fxl\"" Sep 17 17:41:45.178926 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.178827 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb976eb3-1765-41b5-a659-2e3b4c46c7c1-cert\") pod \"ingress-canary-nb9lq\" (UID: \"cb976eb3-1765-41b5-a659-2e3b4c46c7c1\") " pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.178926 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.178873 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmq7\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-kube-api-access-jpmq7\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.178926 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.178898 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54af9832-412c-4a7d-9503-a7f84b6aafad-ca-trust-extracted\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.178942 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe73488a-e40c-4139-bd07-734d7e7b6cb7-config-volume\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179004 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn74j\" (UniqueName: \"kubernetes.io/projected/fe73488a-e40c-4139-bd07-734d7e7b6cb7-kube-api-access-vn74j\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179025 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe73488a-e40c-4139-bd07-734d7e7b6cb7-tmp-dir\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179042 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-registry-tls\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179070 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54af9832-412c-4a7d-9503-a7f84b6aafad-registry-certificates\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179085 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-bound-sa-token\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179151 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe73488a-e40c-4139-bd07-734d7e7b6cb7-metrics-tls\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.179218 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179205 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54af9832-412c-4a7d-9503-a7f84b6aafad-image-registry-private-configuration\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179601 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179259 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54af9832-412c-4a7d-9503-a7f84b6aafad-trusted-ca\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179601 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179285 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54af9832-412c-4a7d-9503-a7f84b6aafad-installation-pull-secrets\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179601 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179310 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz95z\" (UniqueName: \"kubernetes.io/projected/cb976eb3-1765-41b5-a659-2e3b4c46c7c1-kube-api-access-dz95z\") pod \"ingress-canary-nb9lq\" (UID: \"cb976eb3-1765-41b5-a659-2e3b4c46c7c1\") " pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.179601 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179396 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/54af9832-412c-4a7d-9503-a7f84b6aafad-ca-trust-extracted\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.179884 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.179856 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/54af9832-412c-4a7d-9503-a7f84b6aafad-registry-certificates\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.180199 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.180172 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/54af9832-412c-4a7d-9503-a7f84b6aafad-trusted-ca\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.183119 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.183086 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/54af9832-412c-4a7d-9503-a7f84b6aafad-installation-pull-secrets\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.183214 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.183178 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-registry-tls\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.183293 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.183177 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/54af9832-412c-4a7d-9503-a7f84b6aafad-image-registry-private-configuration\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.189781 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.189752 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmq7\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-kube-api-access-jpmq7\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.189872 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.189755 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54af9832-412c-4a7d-9503-a7f84b6aafad-bound-sa-token\") pod \"image-registry-5bc47694cb-555dv\" (UID: \"54af9832-412c-4a7d-9503-a7f84b6aafad\") " pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.209755 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.209723 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:45.209755 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.209740 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:45.209992 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.209740 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:45.212437 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.212417 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 17:41:45.212559 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.212510 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 17:41:45.212628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.212562 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lspl9\"" Sep 17 17:41:45.212774 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.212752 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jc79n\"" Sep 17 17:41:45.212908 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.212895 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 17:41:45.212976 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.212933 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 17:41:45.280303 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280272 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/64162815-9899-48b9-b6e2-550fd15cff5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.280303 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280317 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz95z\" (UniqueName: \"kubernetes.io/projected/cb976eb3-1765-41b5-a659-2e3b4c46c7c1-kube-api-access-dz95z\") pod \"ingress-canary-nb9lq\" (UID: \"cb976eb3-1765-41b5-a659-2e3b4c46c7c1\") " pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.280605 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb976eb3-1765-41b5-a659-2e3b4c46c7c1-cert\") pod \"ingress-canary-nb9lq\" (UID: \"cb976eb3-1765-41b5-a659-2e3b4c46c7c1\") " pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.280605 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280406 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/64162815-9899-48b9-b6e2-550fd15cff5e-crio-socket\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.280605 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280449 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe73488a-e40c-4139-bd07-734d7e7b6cb7-config-volume\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.280605 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280474 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn74j\" (UniqueName: \"kubernetes.io/projected/fe73488a-e40c-4139-bd07-734d7e7b6cb7-kube-api-access-vn74j\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.280605 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280498 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/64162815-9899-48b9-b6e2-550fd15cff5e-data-volume\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.280605 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280547 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe73488a-e40c-4139-bd07-734d7e7b6cb7-tmp-dir\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.280891 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280667 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/64162815-9899-48b9-b6e2-550fd15cff5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.280891 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280770 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe73488a-e40c-4139-bd07-734d7e7b6cb7-metrics-tls\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.280891 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280808 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvcr\" (UniqueName: \"kubernetes.io/projected/64162815-9899-48b9-b6e2-550fd15cff5e-kube-api-access-xbvcr\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.280891 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.280866 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe73488a-e40c-4139-bd07-734d7e7b6cb7-tmp-dir\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.281070 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.281050 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fe73488a-e40c-4139-bd07-734d7e7b6cb7-config-volume\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.282884 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.282864 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fe73488a-e40c-4139-bd07-734d7e7b6cb7-metrics-tls\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.283026 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.283007 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb976eb3-1765-41b5-a659-2e3b4c46c7c1-cert\") pod \"ingress-canary-nb9lq\" (UID: \"cb976eb3-1765-41b5-a659-2e3b4c46c7c1\") " pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.285835 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.285821 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:45.289497 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.289476 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn74j\" (UniqueName: \"kubernetes.io/projected/fe73488a-e40c-4139-bd07-734d7e7b6cb7-kube-api-access-vn74j\") pod \"dns-default-6f52j\" (UID: \"fe73488a-e40c-4139-bd07-734d7e7b6cb7\") " pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.289610 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.289590 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz95z\" (UniqueName: \"kubernetes.io/projected/cb976eb3-1765-41b5-a659-2e3b4c46c7c1-kube-api-access-dz95z\") pod \"ingress-canary-nb9lq\" (UID: \"cb976eb3-1765-41b5-a659-2e3b4c46c7c1\") " pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.308980 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.308958 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:45.319224 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.319207 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nb9lq" Sep 17 17:41:45.381436 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.381234 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/64162815-9899-48b9-b6e2-550fd15cff5e-crio-socket\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.381436 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.381285 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/64162815-9899-48b9-b6e2-550fd15cff5e-data-volume\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.381436 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.381316 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/64162815-9899-48b9-b6e2-550fd15cff5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.381436 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.381384 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvcr\" (UniqueName: \"kubernetes.io/projected/64162815-9899-48b9-b6e2-550fd15cff5e-kube-api-access-xbvcr\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.381436 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.381431 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/64162815-9899-48b9-b6e2-550fd15cff5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.382872 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.382737 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/64162815-9899-48b9-b6e2-550fd15cff5e-crio-socket\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.382872 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.382791 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/64162815-9899-48b9-b6e2-550fd15cff5e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.382872 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.382808 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/64162815-9899-48b9-b6e2-550fd15cff5e-data-volume\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.384873 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.384837 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/64162815-9899-48b9-b6e2-550fd15cff5e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.400136 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.400067 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvcr\" (UniqueName: \"kubernetes.io/projected/64162815-9899-48b9-b6e2-550fd15cff5e-kube-api-access-xbvcr\") pod \"insights-runtime-extractor-zrbjn\" (UID: \"64162815-9899-48b9-b6e2-550fd15cff5e\") " pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.407850 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.407822 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zrbjn" Sep 17 17:41:45.612296 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.612265 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6f52j"] Sep 17 17:41:45.613235 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.613201 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nb9lq"] Sep 17 17:41:45.613870 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.613847 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bc47694cb-555dv"] Sep 17 17:41:45.615585 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:41:45.615557 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54af9832_412c_4a7d_9503_a7f84b6aafad.slice/crio-6493cf95a6b2f469f07608f86f9178823c460419628fe457645ca57f3dee27cd WatchSource:0}: Error finding container 6493cf95a6b2f469f07608f86f9178823c460419628fe457645ca57f3dee27cd: Status 404 returned error can't find the container with id 6493cf95a6b2f469f07608f86f9178823c460419628fe457645ca57f3dee27cd Sep 17 17:41:45.616100 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:41:45.616076 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb976eb3_1765_41b5_a659_2e3b4c46c7c1.slice/crio-4a81e4eeb457f3718a8b90c0c3eb6e5279a8e2147f26c04f433a54cde88630e1 WatchSource:0}: Error finding container 4a81e4eeb457f3718a8b90c0c3eb6e5279a8e2147f26c04f433a54cde88630e1: Status 404 returned error can't find the container with id 4a81e4eeb457f3718a8b90c0c3eb6e5279a8e2147f26c04f433a54cde88630e1 Sep 17 17:41:45.616782 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:41:45.616762 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe73488a_e40c_4139_bd07_734d7e7b6cb7.slice/crio-f170d9361bd0707d7eb3bbe432063473ab4b1bb14af27464a7840b9396c788ea WatchSource:0}: Error finding container f170d9361bd0707d7eb3bbe432063473ab4b1bb14af27464a7840b9396c788ea: Status 404 returned error can't find the container with id f170d9361bd0707d7eb3bbe432063473ab4b1bb14af27464a7840b9396c788ea Sep 17 17:41:45.693022 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:45.692987 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zrbjn"] Sep 17 17:41:45.696462 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:41:45.696437 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64162815_9899_48b9_b6e2_550fd15cff5e.slice/crio-3c4f1d5256481f2d1cfa1ce0f2780e0065b1a0164f639d4380fee4e0b8adb01a WatchSource:0}: Error finding container 3c4f1d5256481f2d1cfa1ce0f2780e0065b1a0164f639d4380fee4e0b8adb01a: Status 404 returned error can't find the container with id 3c4f1d5256481f2d1cfa1ce0f2780e0065b1a0164f639d4380fee4e0b8adb01a Sep 17 17:41:46.522212 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.522173 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc47694cb-555dv" event={"ID":"54af9832-412c-4a7d-9503-a7f84b6aafad","Type":"ContainerStarted","Data":"90d09512b719ae39d98a8e4c1e76ae36887915d9b2f3d4058af365440ed2d590"} Sep 17 17:41:46.522695 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.522236 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bc47694cb-555dv" event={"ID":"54af9832-412c-4a7d-9503-a7f84b6aafad","Type":"ContainerStarted","Data":"6493cf95a6b2f469f07608f86f9178823c460419628fe457645ca57f3dee27cd"} Sep 17 17:41:46.522695 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.522433 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:41:46.524105 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.524074 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nb9lq" event={"ID":"cb976eb3-1765-41b5-a659-2e3b4c46c7c1","Type":"ContainerStarted","Data":"4a81e4eeb457f3718a8b90c0c3eb6e5279a8e2147f26c04f433a54cde88630e1"} Sep 17 17:41:46.525323 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.525298 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6f52j" event={"ID":"fe73488a-e40c-4139-bd07-734d7e7b6cb7","Type":"ContainerStarted","Data":"f170d9361bd0707d7eb3bbe432063473ab4b1bb14af27464a7840b9396c788ea"} Sep 17 17:41:46.527759 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.527737 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrbjn" event={"ID":"64162815-9899-48b9-b6e2-550fd15cff5e","Type":"ContainerStarted","Data":"c80c4bae86607c98d9f570151b19082927a69e5b0b72df2236bcb8dc4b96ae51"} Sep 17 17:41:46.527861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.527790 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrbjn" event={"ID":"64162815-9899-48b9-b6e2-550fd15cff5e","Type":"ContainerStarted","Data":"478ec17219270a82d8e54b48ab8a4a8f46693099c47c92905185790cf54f9b7f"} Sep 17 17:41:46.527861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.527819 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrbjn" event={"ID":"64162815-9899-48b9-b6e2-550fd15cff5e","Type":"ContainerStarted","Data":"3c4f1d5256481f2d1cfa1ce0f2780e0065b1a0164f639d4380fee4e0b8adb01a"} Sep 17 17:41:46.550758 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:46.550698 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bc47694cb-555dv" podStartSLOduration=3.550675554 podStartE2EDuration="3.550675554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:46.550167947 +0000 UTC m=+52.850735491" watchObservedRunningTime="2026-09-17 17:41:46.550675554 +0000 UTC m=+52.851243099" Sep 17 17:41:47.109836 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.109795 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-xq9j8"] Sep 17 17:41:47.111890 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.111864 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-xq9j8" Sep 17 17:41:47.115577 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.115550 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 17:41:47.116047 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.116027 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hblfm\"" Sep 17 17:41:47.116251 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.116232 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 17:41:47.148194 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.148114 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-xq9j8"] Sep 17 17:41:47.298376 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.298330 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkc7r\" (UniqueName: \"kubernetes.io/projected/fa6ce264-24e7-4a2c-8209-1ccff3030d18-kube-api-access-tkc7r\") pod \"downloads-6594f4c75-xq9j8\" (UID: \"fa6ce264-24e7-4a2c-8209-1ccff3030d18\") " pod="openshift-console/downloads-6594f4c75-xq9j8" Sep 17 17:41:47.399188 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.399098 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkc7r\" (UniqueName: \"kubernetes.io/projected/fa6ce264-24e7-4a2c-8209-1ccff3030d18-kube-api-access-tkc7r\") pod \"downloads-6594f4c75-xq9j8\" (UID: \"fa6ce264-24e7-4a2c-8209-1ccff3030d18\") " pod="openshift-console/downloads-6594f4c75-xq9j8" Sep 17 17:41:47.410500 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.410466 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkc7r\" (UniqueName: \"kubernetes.io/projected/fa6ce264-24e7-4a2c-8209-1ccff3030d18-kube-api-access-tkc7r\") pod \"downloads-6594f4c75-xq9j8\" (UID: \"fa6ce264-24e7-4a2c-8209-1ccff3030d18\") " pod="openshift-console/downloads-6594f4c75-xq9j8" Sep 17 17:41:47.423155 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:47.423118 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-xq9j8" Sep 17 17:41:48.863859 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:48.863751 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-xq9j8"] Sep 17 17:41:48.868030 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:41:48.868003 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6ce264_24e7_4a2c_8209_1ccff3030d18.slice/crio-32178b6442de0d32e2ec6aff9dec7477d3f71f71b9fe576c03b3776c0eca1a86 WatchSource:0}: Error finding container 32178b6442de0d32e2ec6aff9dec7477d3f71f71b9fe576c03b3776c0eca1a86: Status 404 returned error can't find the container with id 32178b6442de0d32e2ec6aff9dec7477d3f71f71b9fe576c03b3776c0eca1a86 Sep 17 17:41:49.538977 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.538937 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-xq9j8" event={"ID":"fa6ce264-24e7-4a2c-8209-1ccff3030d18","Type":"ContainerStarted","Data":"32178b6442de0d32e2ec6aff9dec7477d3f71f71b9fe576c03b3776c0eca1a86"} Sep 17 17:41:49.540866 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.540835 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nb9lq" event={"ID":"cb976eb3-1765-41b5-a659-2e3b4c46c7c1","Type":"ContainerStarted","Data":"62ad9fbdb9e386f198896a4f2e5513761a0273772c3ef0ac26ea0fefbb6abde6"} Sep 17 17:41:49.543018 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.542982 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6f52j" event={"ID":"fe73488a-e40c-4139-bd07-734d7e7b6cb7","Type":"ContainerStarted","Data":"ca7ccb7fce13e1036aab7626fba26cee74beb39c4294faf3686bd582d7a3160e"} Sep 17 17:41:49.543177 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.543046 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:49.543177 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.543074 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6f52j" event={"ID":"fe73488a-e40c-4139-bd07-734d7e7b6cb7","Type":"ContainerStarted","Data":"1049c540ba1c94e82ed2151e579c46c5402a837c6e0b0949b80596ab35ee99f4"} Sep 17 17:41:49.545712 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.545680 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zrbjn" event={"ID":"64162815-9899-48b9-b6e2-550fd15cff5e","Type":"ContainerStarted","Data":"4aca6bd505bae263b37d247f30cd9ee38d887850fd46d2c752c5808cdb2c766a"} Sep 17 17:41:49.558697 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.558632 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nb9lq" podStartSLOduration=2.582610393 podStartE2EDuration="5.558609068s" totalImagesPullingTime="2.975998675s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:49.558439183 +0000 UTC m=+55.859006730" watchObservedRunningTime="2026-09-17 17:41:49.558609068 +0000 UTC m=+55.859176610" Sep 17 17:41:49.580334 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:49.580269 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6f52j" podStartSLOduration=2.61633321 podStartE2EDuration="5.580247774s" totalImagesPullingTime="2.963914564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:49.579570918 +0000 UTC m=+55.880138461" watchObservedRunningTime="2026-09-17 17:41:49.580247774 +0000 UTC m=+55.880815319" Sep 17 17:41:54.469739 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:54.469701 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fzxzd" Sep 17 17:41:54.516242 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:54.516179 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zrbjn" podStartSLOduration=6.763352278 podStartE2EDuration="9.516158982s" totalImagesPullingTime="2.752806704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:41:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:41:49.61039361 +0000 UTC m=+55.910961155" watchObservedRunningTime="2026-09-17 17:41:54.516158982 +0000 UTC m=+60.816726527" Sep 17 17:41:59.555061 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.555030 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6f52j" Sep 17 17:41:59.906018 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.905920 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:59.906018 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.905977 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:59.906018 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.906013 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:59.909070 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.909040 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 17:41:59.909243 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.909039 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 17:41:59.909243 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.909054 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 17:41:59.919363 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.919303 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 17:41:59.919569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.919379 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1eca5fda-05d1-43bf-8b60-a65f9480bcc3-original-pull-secret\") pod \"global-pull-secret-syncer-g7gkl\" (UID: \"1eca5fda-05d1-43bf-8b60-a65f9480bcc3\") " pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:59.919569 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.919431 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7996a81-8ee5-41db-a2a0-e3a1f6b13367-metrics-certs\") pod \"network-metrics-daemon-gg8ft\" (UID: \"a7996a81-8ee5-41db-a2a0-e3a1f6b13367\") " pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:41:59.925079 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.925049 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g7gkl" Sep 17 17:41:59.930114 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.930087 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpzc\" (UniqueName: \"kubernetes.io/projected/485f436b-3de3-4aeb-aec2-702f81e96c0a-kube-api-access-qlpzc\") pod \"network-check-target-dm5ls\" (UID: \"485f436b-3de3-4aeb-aec2-702f81e96c0a\") " pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:41:59.932151 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.932132 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jc79n\"" Sep 17 17:41:59.940321 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:41:59.940294 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gg8ft" Sep 17 17:42:00.222147 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:00.222115 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lspl9\"" Sep 17 17:42:00.229759 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:00.229715 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:42:04.166417 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.166316 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw"] Sep 17 17:42:04.168489 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.168466 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.173660 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.173636 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 17:42:04.173810 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.173675 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 17 17:42:04.173810 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.173636 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 17:42:04.173810 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.173750 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-glq9s\"" Sep 17 17:42:04.173977 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.173636 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 17 17:42:04.174129 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.174076 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 17:42:04.191328 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.191303 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw"] Sep 17 17:42:04.237544 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.237488 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.237544 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.237548 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2db7991c-a774-40cd-8aa4-57a753d33734-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.237769 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.237613 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.237769 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.237645 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdwqq\" (UniqueName: \"kubernetes.io/projected/2db7991c-a774-40cd-8aa4-57a753d33734-kube-api-access-zdwqq\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.242046 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.242020 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4ksjc"] Sep 17 17:42:04.244591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.244553 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.250781 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.250742 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 17:42:04.251078 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.251057 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-x5mb7\"" Sep 17 17:42:04.251199 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.251107 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 17:42:04.251351 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.251328 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338012 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce6dba-083e-4b97-b623-afe19826b0fe-metrics-client-ca\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338069 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338111 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338138 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2db7991c-a774-40cd-8aa4-57a753d33734-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338181 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-sys\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338251 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-tls\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338302 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-root\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338328 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.338419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338409 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvms\" (UniqueName: \"kubernetes.io/projected/e4ce6dba-083e-4b97-b623-afe19826b0fe-kube-api-access-8fvms\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.339001 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338467 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-wtmp\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.339001 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338584 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-textfile\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.339001 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338645 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.339001 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.338682 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdwqq\" (UniqueName: \"kubernetes.io/projected/2db7991c-a774-40cd-8aa4-57a753d33734-kube-api-access-zdwqq\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.339001 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:42:04.338812 2550 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 17 17:42:04.339001 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:42:04.338904 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-tls podName:2db7991c-a774-40cd-8aa4-57a753d33734 nodeName:}" failed. No retries permitted until 2026-09-17 17:42:04.83887793 +0000 UTC m=+71.139445464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-ns2kw" (UID: "2db7991c-a774-40cd-8aa4-57a753d33734") : secret "openshift-state-metrics-tls" not found Sep 17 17:42:04.339299 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.339230 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2db7991c-a774-40cd-8aa4-57a753d33734-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.344308 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.344111 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.360500 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.360476 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdwqq\" (UniqueName: \"kubernetes.io/projected/2db7991c-a774-40cd-8aa4-57a753d33734-kube-api-access-zdwqq\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.439055 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.438976 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-sys\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439055 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439016 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-tls\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439055 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439051 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-root\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439077 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439105 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvms\" (UniqueName: \"kubernetes.io/projected/e4ce6dba-083e-4b97-b623-afe19826b0fe-kube-api-access-8fvms\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439078 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-sys\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439148 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-wtmp\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439179 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-root\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439203 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-textfile\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439247 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce6dba-083e-4b97-b623-afe19826b0fe-metrics-client-ca\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439275 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439271 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439664 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439292 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-wtmp\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439664 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439564 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-textfile\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439812 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439785 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.439875 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.439829 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4ce6dba-083e-4b97-b623-afe19826b0fe-metrics-client-ca\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.441326 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.441307 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-tls\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.441465 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.441448 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4ce6dba-083e-4b97-b623-afe19826b0fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.461147 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.461126 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvms\" (UniqueName: \"kubernetes.io/projected/e4ce6dba-083e-4b97-b623-afe19826b0fe-kube-api-access-8fvms\") pod \"node-exporter-4ksjc\" (UID: \"e4ce6dba-083e-4b97-b623-afe19826b0fe\") " pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.550979 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.550917 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gg8ft"] Sep 17 17:42:04.553890 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:04.553853 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7996a81_8ee5_41db_a2a0_e3a1f6b13367.slice/crio-bb2a21d187251077b0548227fcfd69a30f36a1d6cb99071ad561afce9e7fbb3c WatchSource:0}: Error finding container bb2a21d187251077b0548227fcfd69a30f36a1d6cb99071ad561afce9e7fbb3c: Status 404 returned error can't find the container with id bb2a21d187251077b0548227fcfd69a30f36a1d6cb99071ad561afce9e7fbb3c Sep 17 17:42:04.556342 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.556104 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4ksjc" Sep 17 17:42:04.565019 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.564576 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g7gkl"] Sep 17 17:42:04.567544 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:04.567507 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4ce6dba_083e_4b97_b623_afe19826b0fe.slice/crio-0a9b86105b15b1cdc5fec132ba5c947f8a10a9a7bdb40b7307f12dae405e6446 WatchSource:0}: Error finding container 0a9b86105b15b1cdc5fec132ba5c947f8a10a9a7bdb40b7307f12dae405e6446: Status 404 returned error can't find the container with id 0a9b86105b15b1cdc5fec132ba5c947f8a10a9a7bdb40b7307f12dae405e6446 Sep 17 17:42:04.568681 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:04.568656 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eca5fda_05d1_43bf_8b60_a65f9480bcc3.slice/crio-c43c06baf88eb17e2c62970c5cc961fac333d20bdb01b262ab887931a2436525 WatchSource:0}: Error finding container c43c06baf88eb17e2c62970c5cc961fac333d20bdb01b262ab887931a2436525: Status 404 returned error can't find the container with id c43c06baf88eb17e2c62970c5cc961fac333d20bdb01b262ab887931a2436525 Sep 17 17:42:04.587893 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.587865 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dm5ls"] Sep 17 17:42:04.591422 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:04.591389 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485f436b_3de3_4aeb_aec2_702f81e96c0a.slice/crio-34f163600bf1ddab401418286c3c7d11d776a8a7921aad963a31e5641d2372b7 WatchSource:0}: Error finding container 34f163600bf1ddab401418286c3c7d11d776a8a7921aad963a31e5641d2372b7: Status 404 returned error can't find the container with id 34f163600bf1ddab401418286c3c7d11d776a8a7921aad963a31e5641d2372b7 Sep 17 17:42:04.601949 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.601921 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-xq9j8" event={"ID":"fa6ce264-24e7-4a2c-8209-1ccff3030d18","Type":"ContainerStarted","Data":"9c8d7fb3327a396aa6c75f243b0c7ab83f4c7d2e572a191cf89c0900bde3b1e8"} Sep 17 17:42:04.602603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.602557 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-xq9j8" Sep 17 17:42:04.603466 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.603439 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gg8ft" event={"ID":"a7996a81-8ee5-41db-a2a0-e3a1f6b13367","Type":"ContainerStarted","Data":"bb2a21d187251077b0548227fcfd69a30f36a1d6cb99071ad561afce9e7fbb3c"} Sep 17 17:42:04.604882 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.604861 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7gkl" event={"ID":"1eca5fda-05d1-43bf-8b60-a65f9480bcc3","Type":"ContainerStarted","Data":"c43c06baf88eb17e2c62970c5cc961fac333d20bdb01b262ab887931a2436525"} Sep 17 17:42:04.606521 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.606480 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ksjc" event={"ID":"e4ce6dba-083e-4b97-b623-afe19826b0fe","Type":"ContainerStarted","Data":"0a9b86105b15b1cdc5fec132ba5c947f8a10a9a7bdb40b7307f12dae405e6446"} Sep 17 17:42:04.607625 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.607602 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dm5ls" event={"ID":"485f436b-3de3-4aeb-aec2-702f81e96c0a","Type":"ContainerStarted","Data":"34f163600bf1ddab401418286c3c7d11d776a8a7921aad963a31e5641d2372b7"} Sep 17 17:42:04.624580 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.624553 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-xq9j8" Sep 17 17:42:04.626681 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.626610 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-xq9j8" podStartSLOduration=2.142049667 podStartE2EDuration="17.626594631s" totalImagesPullingTime="15.484544964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:41:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:04.62558285 +0000 UTC m=+70.926150395" watchObservedRunningTime="2026-09-17 17:42:04.626594631 +0000 UTC m=+70.927162169" Sep 17 17:42:04.841917 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.841834 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:04.844691 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:04.844666 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2db7991c-a774-40cd-8aa4-57a753d33734-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-ns2kw\" (UID: \"2db7991c-a774-40cd-8aa4-57a753d33734\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:05.083102 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.083060 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" Sep 17 17:42:05.407598 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.402777 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw"] Sep 17 17:42:05.415980 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:05.415938 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db7991c_a774_40cd_8aa4_57a753d33734.slice/crio-0d427821a42c14ca9e95323dc2f15901d730b1feaa1fdb5ac0dd12df16ee0c72 WatchSource:0}: Error finding container 0d427821a42c14ca9e95323dc2f15901d730b1feaa1fdb5ac0dd12df16ee0c72: Status 404 returned error can't find the container with id 0d427821a42c14ca9e95323dc2f15901d730b1feaa1fdb5ac0dd12df16ee0c72 Sep 17 17:42:05.478950 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.477071 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:42:05.480685 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.480651 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.485775 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.485692 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 17:42:05.486106 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.485986 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 17:42:05.487785 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.487662 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 17:42:05.488871 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.487921 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rctvc\"" Sep 17 17:42:05.488871 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.488147 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 17:42:05.488871 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.488163 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 17:42:05.488871 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.488366 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 17:42:05.488871 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.488445 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 17:42:05.488871 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.488576 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 17:42:05.488871 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.488686 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 17:42:05.505934 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.505808 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548170 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-web-config\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548233 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548265 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548322 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548389 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-volume\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548416 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548441 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk6bb\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-kube-api-access-fk6bb\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.548603 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.548467 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.550250 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.550039 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.550250 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.550081 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.550250 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.550114 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-out\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.550250 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.550140 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.615263 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.615223 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" event={"ID":"2db7991c-a774-40cd-8aa4-57a753d33734","Type":"ContainerStarted","Data":"2c81778963fb7d2d2bafa46c0235c71ebaecc2b36ef7cab562f58d2a2e5134bf"} Sep 17 17:42:05.615444 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.615288 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" event={"ID":"2db7991c-a774-40cd-8aa4-57a753d33734","Type":"ContainerStarted","Data":"0d427821a42c14ca9e95323dc2f15901d730b1feaa1fdb5ac0dd12df16ee0c72"} Sep 17 17:42:05.651444 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651406 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651455 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk6bb\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-kube-api-access-fk6bb\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651488 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651514 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651555 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651582 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-out\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651605 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651651 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651652 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-web-config\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651990 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651705 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651990 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651731 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651990 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651758 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651990 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651793 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.651990 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.651822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-volume\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.653328 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.653000 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.656964 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:42:05.654952 2550 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 17 17:42:05.656964 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:42:05.655018 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls podName:df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa nodeName:}" failed. No retries permitted until 2026-09-17 17:42:06.154997782 +0000 UTC m=+72.455565310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa") : secret "alertmanager-main-tls" not found Sep 17 17:42:05.656964 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.656906 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-volume\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.657257 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.656965 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.657378 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.657340 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.657567 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.657497 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.668643 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.662461 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-out\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.668643 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.665080 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.668643 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.665566 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.668643 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.665652 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.669798 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.669741 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.670747 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.670691 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-web-config\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:05.672869 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:05.672642 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk6bb\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-kube-api-access-fk6bb\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:06.157468 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:06.157402 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:06.164699 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:06.164666 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:06.397257 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:06.397167 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:42:07.464243 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.463556 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:42:07.502705 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:07.502659 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf249ff7_b4bc_4e8e_8e4c_6ba6a6e84efa.slice/crio-7318282030af8c253013c0b2e570954ce851cf9ab42a48324b41c15209879784 WatchSource:0}: Error finding container 7318282030af8c253013c0b2e570954ce851cf9ab42a48324b41c15209879784: Status 404 returned error can't find the container with id 7318282030af8c253013c0b2e570954ce851cf9ab42a48324b41c15209879784 Sep 17 17:42:07.537811 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.537145 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bc47694cb-555dv" Sep 17 17:42:07.629672 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.629577 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerStarted","Data":"7318282030af8c253013c0b2e570954ce851cf9ab42a48324b41c15209879784"} Sep 17 17:42:07.632803 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.632768 2550 generic.go:356] "Generic (PLEG): container finished" podID="e4ce6dba-083e-4b97-b623-afe19826b0fe" containerID="5c3cacee5a897b214420005f9877ed6b1b8663ce449f7be46414136dd05da82f" exitCode=0 Sep 17 17:42:07.633084 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.633059 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ksjc" event={"ID":"e4ce6dba-083e-4b97-b623-afe19826b0fe","Type":"ContainerDied","Data":"5c3cacee5a897b214420005f9877ed6b1b8663ce449f7be46414136dd05da82f"} Sep 17 17:42:07.637313 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.637226 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" event={"ID":"2db7991c-a774-40cd-8aa4-57a753d33734","Type":"ContainerStarted","Data":"f58c7a92122ce49545035032cd752b3ecfcec2ed028d56ea8e99547a22fafd7c"} Sep 17 17:42:07.650711 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.647932 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gg8ft" event={"ID":"a7996a81-8ee5-41db-a2a0-e3a1f6b13367","Type":"ContainerStarted","Data":"80aeff551b59df9df63f8d80d7b036ce2787b484e82b90ef518d46aa163f1b1a"} Sep 17 17:42:07.650711 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.647997 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gg8ft" event={"ID":"a7996a81-8ee5-41db-a2a0-e3a1f6b13367","Type":"ContainerStarted","Data":"f16e245993acb83be4ae93fd00dbbf9ac9a693386a37d3f9ccbfe2256edb5e14"} Sep 17 17:42:07.774906 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:07.773502 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gg8ft" podStartSLOduration=70.218529109 podStartE2EDuration="1m12.773480591s" totalImagesPullingTime="2.554951482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:07.771830913 +0000 UTC m=+74.072398502" watchObservedRunningTime="2026-09-17 17:42:07.773480591 +0000 UTC m=+74.074048136" Sep 17 17:42:08.667487 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.667372 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ksjc" event={"ID":"e4ce6dba-083e-4b97-b623-afe19826b0fe","Type":"ContainerStarted","Data":"67a939e14d68d56634faf06b2ceac63e1cb63fc17dd8512e512b7020a230b4cf"} Sep 17 17:42:08.667487 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.667440 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4ksjc" event={"ID":"e4ce6dba-083e-4b97-b623-afe19826b0fe","Type":"ContainerStarted","Data":"fbe2ffac1158a9137d35689d75a59cdefef60c6b0d7feb588acf3c0486cdb890"} Sep 17 17:42:08.690310 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.690175 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4ksjc" podStartSLOduration=2.031410459 podStartE2EDuration="4.689819346s" totalImagesPullingTime="2.554676693s" totalInitContainerRuntime="103.732194ms" isStatefulPod=true podCreationTimestamp="2026-09-17 17:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:08.688270731 +0000 UTC m=+74.988838276" watchObservedRunningTime="2026-09-17 17:42:08.689819346 +0000 UTC m=+74.990386894" Sep 17 17:42:08.706518 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.705267 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d4b6d4d65-62sjd"] Sep 17 17:42:08.724820 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.724771 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d4b6d4d65-62sjd"] Sep 17 17:42:08.726546 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.725520 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.728922 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.728832 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d4zb2\"" Sep 17 17:42:08.729171 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.729105 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 17 17:42:08.729998 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.729977 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 17 17:42:08.730264 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.730244 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 17:42:08.730799 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.730507 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 17 17:42:08.730799 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.730646 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c59gs4jmq0mj1\"" Sep 17 17:42:08.784786 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.784746 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.785243 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.785216 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-client-ca-bundle\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.785347 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.785300 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-secret-metrics-server-tls\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.785347 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.785334 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-metrics-server-audit-profiles\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.785463 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.785378 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-audit-log\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.785463 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.785423 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q287s\" (UniqueName: \"kubernetes.io/projected/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-kube-api-access-q287s\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.785589 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.785512 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-secret-metrics-server-client-certs\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.887024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.886811 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-client-ca-bundle\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.887024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.886877 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-secret-metrics-server-tls\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.887024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.886904 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-metrics-server-audit-profiles\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.887024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.886937 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-audit-log\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.887024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.886962 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q287s\" (UniqueName: \"kubernetes.io/projected/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-kube-api-access-q287s\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.887024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.887018 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-secret-metrics-server-client-certs\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.887427 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.887046 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.888014 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.887987 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.890463 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.888664 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-audit-log\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.890463 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.889856 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-metrics-server-audit-profiles\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.900144 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.900053 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-client-ca-bundle\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.901194 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.901165 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q287s\" (UniqueName: \"kubernetes.io/projected/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-kube-api-access-q287s\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.902704 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.902660 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-secret-metrics-server-client-certs\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.904643 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.904596 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3-secret-metrics-server-tls\") pod \"metrics-server-5d4b6d4d65-62sjd\" (UID: \"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3\") " pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:08.962744 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.962652 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g"] Sep 17 17:42:08.997604 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.997564 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g"] Sep 17 17:42:08.997782 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:08.997680 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:09.002377 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.002349 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 17 17:42:09.009070 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.008592 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6rv7d\"" Sep 17 17:42:09.041179 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.041128 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:09.089459 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.089419 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bca4d02-f77a-40e6-9f71-7f05742facd6-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dwk4g\" (UID: \"8bca4d02-f77a-40e6-9f71-7f05742facd6\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:09.089727 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.089467 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8msjc\" (UniqueName: \"kubernetes.io/projected/8bca4d02-f77a-40e6-9f71-7f05742facd6-kube-api-access-8msjc\") pod \"monitoring-plugin-5984459ff8-dwk4g\" (UID: \"8bca4d02-f77a-40e6-9f71-7f05742facd6\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:09.190065 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.190021 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bca4d02-f77a-40e6-9f71-7f05742facd6-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dwk4g\" (UID: \"8bca4d02-f77a-40e6-9f71-7f05742facd6\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:09.190065 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.190073 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8msjc\" (UniqueName: \"kubernetes.io/projected/8bca4d02-f77a-40e6-9f71-7f05742facd6-kube-api-access-8msjc\") pod \"monitoring-plugin-5984459ff8-dwk4g\" (UID: \"8bca4d02-f77a-40e6-9f71-7f05742facd6\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:09.193577 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.193498 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8bca4d02-f77a-40e6-9f71-7f05742facd6-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dwk4g\" (UID: \"8bca4d02-f77a-40e6-9f71-7f05742facd6\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:09.200861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.200800 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8msjc\" (UniqueName: \"kubernetes.io/projected/8bca4d02-f77a-40e6-9f71-7f05742facd6-kube-api-access-8msjc\") pod \"monitoring-plugin-5984459ff8-dwk4g\" (UID: \"8bca4d02-f77a-40e6-9f71-7f05742facd6\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:09.314865 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:09.314499 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:13.135735 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.135704 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d4b6d4d65-62sjd"] Sep 17 17:42:13.139797 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:13.139770 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde5a7cc_f485_49aa_8fca_06eaf2b5b6f3.slice/crio-8044caede52d776fad84760216567b022951fb9f69924eafe009b86aab3c16b0 WatchSource:0}: Error finding container 8044caede52d776fad84760216567b022951fb9f69924eafe009b86aab3c16b0: Status 404 returned error can't find the container with id 8044caede52d776fad84760216567b022951fb9f69924eafe009b86aab3c16b0 Sep 17 17:42:13.152637 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.152565 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g"] Sep 17 17:42:13.156713 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:42:13.156686 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bca4d02_f77a_40e6_9f71_7f05742facd6.slice/crio-e0e2b6eb9e3cb2f7e5565224c2f3c7997b2b7f51f1b1fe6c2282d6acfbd69ac1 WatchSource:0}: Error finding container e0e2b6eb9e3cb2f7e5565224c2f3c7997b2b7f51f1b1fe6c2282d6acfbd69ac1: Status 404 returned error can't find the container with id e0e2b6eb9e3cb2f7e5565224c2f3c7997b2b7f51f1b1fe6c2282d6acfbd69ac1 Sep 17 17:42:13.687742 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.687702 2550 generic.go:356] "Generic (PLEG): container finished" podID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerID="fe44aa93ebf3d19101403e5fcc3816b8c9c9d18a79542b6674260df34c33db10" exitCode=0 Sep 17 17:42:13.687920 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.687790 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"fe44aa93ebf3d19101403e5fcc3816b8c9c9d18a79542b6674260df34c33db10"} Sep 17 17:42:13.689099 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.689079 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" event={"ID":"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3","Type":"ContainerStarted","Data":"8044caede52d776fad84760216567b022951fb9f69924eafe009b86aab3c16b0"} Sep 17 17:42:13.690594 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.690572 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dm5ls" event={"ID":"485f436b-3de3-4aeb-aec2-702f81e96c0a","Type":"ContainerStarted","Data":"42ad7d09993dd9ce09965018ccc2c63a6dde0d51e8945c8e4f62e901ee7b1304"} Sep 17 17:42:13.690703 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.690689 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:42:13.692918 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.692894 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" event={"ID":"2db7991c-a774-40cd-8aa4-57a753d33734","Type":"ContainerStarted","Data":"56bda488a9205bcdf64f33cc42aaff3ae19b65b71cc1b2f81149e9a0213283c4"} Sep 17 17:42:13.694079 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.694056 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" event={"ID":"8bca4d02-f77a-40e6-9f71-7f05742facd6","Type":"ContainerStarted","Data":"e0e2b6eb9e3cb2f7e5565224c2f3c7997b2b7f51f1b1fe6c2282d6acfbd69ac1"} Sep 17 17:42:13.695505 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.695487 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g7gkl" event={"ID":"1eca5fda-05d1-43bf-8b60-a65f9480bcc3","Type":"ContainerStarted","Data":"ff1ff36918bebac5d973216a89f6171c9665ea949f54c9c977cc122296cf2cac"} Sep 17 17:42:13.745359 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.745295 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-ns2kw" podStartSLOduration=4.178013799 podStartE2EDuration="9.745272851s" totalImagesPullingTime="5.567259052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:13.743890916 +0000 UTC m=+80.044458493" watchObservedRunningTime="2026-09-17 17:42:13.745272851 +0000 UTC m=+80.045840397" Sep 17 17:42:13.763150 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.763078 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g7gkl" podStartSLOduration=70.463818877 podStartE2EDuration="1m18.763056412s" totalImagesPullingTime="8.299237535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:13.761937069 +0000 UTC m=+80.062504614" watchObservedRunningTime="2026-09-17 17:42:13.763056412 +0000 UTC m=+80.063623956" Sep 17 17:42:13.782323 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:13.782252 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dm5ls" podStartSLOduration=70.522700603 podStartE2EDuration="1m18.782228339s" totalImagesPullingTime="8.259527736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:13.781491841 +0000 UTC m=+80.082059401" watchObservedRunningTime="2026-09-17 17:42:13.782228339 +0000 UTC m=+80.082795886" Sep 17 17:42:16.716246 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.716213 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerStarted","Data":"d9f59ae20a40115c80926a5a3fb4ecc136c40ed5a2120a527c8c2fffe5916d33"} Sep 17 17:42:16.716650 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.716297 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerStarted","Data":"0f14f62c58149bfc1429f1deb1deb03f21c4cf5518a2c0e5038b718ffd4c3682"} Sep 17 17:42:16.716650 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.716346 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerStarted","Data":"ce7bf07b387f359512e7dddd949c94bfa70f5da9925106a25e7c2a9f17823f74"} Sep 17 17:42:16.716650 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.716386 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerStarted","Data":"68c673ba9eac4b0bc56f03db9499d12bfa98d9307d33f1663d825b6312575e18"} Sep 17 17:42:16.716650 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.716432 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerStarted","Data":"a80dfc970b0cb7793ab5ba1ec6f74802fa18f19134d0a9e8c8c25177ec9f9739"} Sep 17 17:42:16.717912 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.717884 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" event={"ID":"cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3","Type":"ContainerStarted","Data":"897ebae06deb170456f47a1ace17d954923c4cd15ce21f4db5fa0fc376777254"} Sep 17 17:42:16.719412 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.719385 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" event={"ID":"8bca4d02-f77a-40e6-9f71-7f05742facd6","Type":"ContainerStarted","Data":"baee0e8044b195e61cb456ad36f2f0e46b04ee12e115e14b80a87abef6d7dca5"} Sep 17 17:42:16.719611 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.719591 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:16.724061 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.724031 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" Sep 17 17:42:16.740361 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.740313 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" podStartSLOduration=5.714159327 podStartE2EDuration="8.740295906s" totalImagesPullingTime="3.026136579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:16.738606361 +0000 UTC m=+83.039173906" watchObservedRunningTime="2026-09-17 17:42:16.740295906 +0000 UTC m=+83.040863450" Sep 17 17:42:16.759154 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:16.759093 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dwk4g" podStartSLOduration=5.750369424 podStartE2EDuration="8.759069717s" totalImagesPullingTime="3.008700293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:16.757039022 +0000 UTC m=+83.057606566" watchObservedRunningTime="2026-09-17 17:42:16.759069717 +0000 UTC m=+83.059637262" Sep 17 17:42:18.733435 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:18.733388 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerStarted","Data":"2a6a8dc54c5b00635e7fb82a4c7d76193bc10a7457f5786666c9941d1676bfb8"} Sep 17 17:42:18.774096 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:18.774021 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.879405338 podStartE2EDuration="13.773999879s" totalImagesPullingTime="8.875593234s" totalInitContainerRuntime="19.001307ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:42:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:42:18.771277459 +0000 UTC m=+85.071845007" watchObservedRunningTime="2026-09-17 17:42:18.773999879 +0000 UTC m=+85.074567427" Sep 17 17:42:29.042254 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:29.042210 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:29.042837 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:29.042284 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:44.703331 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:44.703294 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dm5ls" Sep 17 17:42:47.607890 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:47.607856 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/init-config-reloader/0.log" Sep 17 17:42:47.614773 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:47.614732 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/alertmanager/0.log" Sep 17 17:42:47.696917 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:47.696884 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/config-reloader/0.log" Sep 17 17:42:47.897303 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:47.897225 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/kube-rbac-proxy-web/0.log" Sep 17 17:42:48.106145 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:48.106116 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/kube-rbac-proxy/0.log" Sep 17 17:42:48.298278 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:48.298247 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/kube-rbac-proxy-metric/0.log" Sep 17 17:42:48.498830 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:48.498808 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/prom-label-proxy/0.log" Sep 17 17:42:49.047466 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:49.047436 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:49.051346 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:49.051326 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d4b6d4d65-62sjd" Sep 17 17:42:49.499046 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:49.499020 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d4b6d4d65-62sjd_cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3/metrics-server/0.log" Sep 17 17:42:49.697898 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:49.697859 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-dwk4g_8bca4d02-f77a-40e6-9f71-7f05742facd6/monitoring-plugin/0.log" Sep 17 17:42:49.898778 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:49.898710 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ksjc_e4ce6dba-083e-4b97-b623-afe19826b0fe/init-textfile/0.log" Sep 17 17:42:50.098484 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:50.098455 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ksjc_e4ce6dba-083e-4b97-b623-afe19826b0fe/node-exporter/0.log" Sep 17 17:42:50.297456 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:50.297425 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ksjc_e4ce6dba-083e-4b97-b623-afe19826b0fe/kube-rbac-proxy/0.log" Sep 17 17:42:51.697538 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:51.697496 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ns2kw_2db7991c-a774-40cd-8aa4-57a753d33734/kube-rbac-proxy-main/0.log" Sep 17 17:42:51.897779 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:51.897748 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ns2kw_2db7991c-a774-40cd-8aa4-57a753d33734/kube-rbac-proxy-self/0.log" Sep 17 17:42:52.097938 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:52.097869 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ns2kw_2db7991c-a774-40cd-8aa4-57a753d33734/openshift-state-metrics/0.log" Sep 17 17:42:57.099074 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:42:57.099039 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-xq9j8_fa6ce264-24e7-4a2c-8209-1ccff3030d18/download-server/0.log" Sep 17 17:43:24.742697 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.742663 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:43:24.743229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.742934 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="alertmanager" containerID="cri-o://a80dfc970b0cb7793ab5ba1ec6f74802fa18f19134d0a9e8c8c25177ec9f9739" gracePeriod=120 Sep 17 17:43:24.743229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.742995 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-metric" containerID="cri-o://d9f59ae20a40115c80926a5a3fb4ecc136c40ed5a2120a527c8c2fffe5916d33" gracePeriod=120 Sep 17 17:43:24.743229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.743021 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-web" containerID="cri-o://ce7bf07b387f359512e7dddd949c94bfa70f5da9925106a25e7c2a9f17823f74" gracePeriod=120 Sep 17 17:43:24.743229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.743052 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy" containerID="cri-o://0f14f62c58149bfc1429f1deb1deb03f21c4cf5518a2c0e5038b718ffd4c3682" gracePeriod=120 Sep 17 17:43:24.743229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.743083 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="prom-label-proxy" containerID="cri-o://2a6a8dc54c5b00635e7fb82a4c7d76193bc10a7457f5786666c9941d1676bfb8" gracePeriod=120 Sep 17 17:43:24.743229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.743033 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="config-reloader" containerID="cri-o://68c673ba9eac4b0bc56f03db9499d12bfa98d9307d33f1663d825b6312575e18" gracePeriod=120 Sep 17 17:43:24.973983 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.973950 2550 generic.go:356] "Generic (PLEG): container finished" podID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerID="2a6a8dc54c5b00635e7fb82a4c7d76193bc10a7457f5786666c9941d1676bfb8" exitCode=0 Sep 17 17:43:24.973983 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.973971 2550 generic.go:356] "Generic (PLEG): container finished" podID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerID="d9f59ae20a40115c80926a5a3fb4ecc136c40ed5a2120a527c8c2fffe5916d33" exitCode=0 Sep 17 17:43:24.973983 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.973977 2550 generic.go:356] "Generic (PLEG): container finished" podID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerID="0f14f62c58149bfc1429f1deb1deb03f21c4cf5518a2c0e5038b718ffd4c3682" exitCode=0 Sep 17 17:43:24.973983 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.973983 2550 generic.go:356] "Generic (PLEG): container finished" podID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerID="68c673ba9eac4b0bc56f03db9499d12bfa98d9307d33f1663d825b6312575e18" exitCode=0 Sep 17 17:43:24.973983 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.973994 2550 generic.go:356] "Generic (PLEG): container finished" podID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerID="a80dfc970b0cb7793ab5ba1ec6f74802fa18f19134d0a9e8c8c25177ec9f9739" exitCode=0 Sep 17 17:43:24.974277 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.974026 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"2a6a8dc54c5b00635e7fb82a4c7d76193bc10a7457f5786666c9941d1676bfb8"} Sep 17 17:43:24.974277 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.974100 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"d9f59ae20a40115c80926a5a3fb4ecc136c40ed5a2120a527c8c2fffe5916d33"} Sep 17 17:43:24.974277 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.974126 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"0f14f62c58149bfc1429f1deb1deb03f21c4cf5518a2c0e5038b718ffd4c3682"} Sep 17 17:43:24.974277 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.974150 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"68c673ba9eac4b0bc56f03db9499d12bfa98d9307d33f1663d825b6312575e18"} Sep 17 17:43:24.974277 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:24.974173 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"a80dfc970b0cb7793ab5ba1ec6f74802fa18f19134d0a9e8c8c25177ec9f9739"} Sep 17 17:43:25.982278 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:25.982241 2550 generic.go:356] "Generic (PLEG): container finished" podID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerID="ce7bf07b387f359512e7dddd949c94bfa70f5da9925106a25e7c2a9f17823f74" exitCode=0 Sep 17 17:43:25.982722 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:25.982298 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"ce7bf07b387f359512e7dddd949c94bfa70f5da9925106a25e7c2a9f17823f74"} Sep 17 17:43:25.982722 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:25.982367 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa","Type":"ContainerDied","Data":"7318282030af8c253013c0b2e570954ce851cf9ab42a48324b41c15209879784"} Sep 17 17:43:25.982722 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:25.982393 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7318282030af8c253013c0b2e570954ce851cf9ab42a48324b41c15209879784" Sep 17 17:43:25.986776 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:25.986751 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:26.094049 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.093968 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-web-config\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094049 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094010 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-tls-assets\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094049 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094028 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-metrics-client-ca\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094049 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094051 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk6bb\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-kube-api-access-fk6bb\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094074 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-cluster-tls-config\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094105 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-metric\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094130 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094159 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-web\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094202 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-volume\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094235 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-out\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094261 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094300 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-main-db\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094355 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-trusted-ca-bundle\") pod \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\" (UID: \"df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa\") " Sep 17 17:43:26.094833 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094432 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:43:26.094833 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094749 2550 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-metrics-client-ca\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.094833 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.094806 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:43:26.095210 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.095156 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:43:26.097103 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.097070 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-kube-api-access-fk6bb" (OuterVolumeSpecName: "kube-api-access-fk6bb") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "kube-api-access-fk6bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:43:26.097610 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.097363 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:43:26.097610 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.097379 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:43:26.097776 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.097721 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:43:26.097843 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.097793 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-out" (OuterVolumeSpecName: "config-out") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:43:26.098049 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.098016 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-volume" (OuterVolumeSpecName: "config-volume") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:43:26.099274 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.099248 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:43:26.099365 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.099328 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:43:26.101151 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.101123 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:43:26.107306 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.107272 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-web-config" (OuterVolumeSpecName: "web-config") pod "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" (UID: "df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:43:26.196035 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.195991 2550 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-web-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196035 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196024 2550 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-tls-assets\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196035 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196034 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk6bb\" (UniqueName: \"kubernetes.io/projected/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-kube-api-access-fk6bb\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196035 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196043 2550 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-cluster-tls-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196054 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196063 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196072 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196080 2550 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-volume\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196090 2550 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-config-out\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196097 2550 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-secret-alertmanager-main-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196105 2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-main-db\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.196292 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.196113 2550 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:43:26.985315 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:26.985285 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.008679 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.008634 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:43:27.013652 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.013625 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:43:27.040725 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040696 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:43:27.040936 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040924 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="init-config-reloader" Sep 17 17:43:27.040977 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040937 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="init-config-reloader" Sep 17 17:43:27.040977 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040946 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="prom-label-proxy" Sep 17 17:43:27.040977 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040976 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="prom-label-proxy" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040986 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040991 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.040997 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-metric" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041003 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-metric" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041017 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="alertmanager" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041022 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="alertmanager" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041028 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="config-reloader" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041032 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="config-reloader" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041042 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-web" Sep 17 17:43:27.041068 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041048 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-web" Sep 17 17:43:27.041343 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041087 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="config-reloader" Sep 17 17:43:27.041343 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041093 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-metric" Sep 17 17:43:27.041343 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041099 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="alertmanager" Sep 17 17:43:27.041343 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041106 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy-web" Sep 17 17:43:27.041343 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041112 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="prom-label-proxy" Sep 17 17:43:27.041343 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.041117 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" containerName="kube-rbac-proxy" Sep 17 17:43:27.044111 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.044096 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.046798 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046770 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 17:43:27.046955 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046808 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 17:43:27.046955 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046770 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 17:43:27.046955 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046845 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 17:43:27.046955 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046811 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 17:43:27.046955 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046916 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 17:43:27.046955 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046772 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rctvc\"" Sep 17 17:43:27.047229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.046922 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 17:43:27.047229 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.047196 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 17:43:27.052626 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.052607 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 17:43:27.060934 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.060857 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:43:27.102944 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.102910 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.102944 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.102951 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1c434b-dcac-4386-aaf2-4d8e3273de33-config-out\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103113 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.102969 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103113 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.102999 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1c434b-dcac-4386-aaf2-4d8e3273de33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103113 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103023 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103113 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103038 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1c434b-dcac-4386-aaf2-4d8e3273de33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103113 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103065 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1c434b-dcac-4386-aaf2-4d8e3273de33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103113 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103084 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42797\" (UniqueName: \"kubernetes.io/projected/8e1c434b-dcac-4386-aaf2-4d8e3273de33-kube-api-access-42797\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103319 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103113 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-web-config\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103319 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103138 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103319 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103153 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103319 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103190 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1c434b-dcac-4386-aaf2-4d8e3273de33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.103319 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.103212 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204015 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.203978 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1c434b-dcac-4386-aaf2-4d8e3273de33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204015 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204015 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204172 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204038 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204172 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204065 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1c434b-dcac-4386-aaf2-4d8e3273de33-config-out\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204172 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204087 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204172 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204118 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1c434b-dcac-4386-aaf2-4d8e3273de33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204172 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204148 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204172 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204168 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1c434b-dcac-4386-aaf2-4d8e3273de33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204195 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1c434b-dcac-4386-aaf2-4d8e3273de33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204217 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42797\" (UniqueName: \"kubernetes.io/projected/8e1c434b-dcac-4386-aaf2-4d8e3273de33-kube-api-access-42797\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204249 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-web-config\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204276 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204300 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.204455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.204449 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8e1c434b-dcac-4386-aaf2-4d8e3273de33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.205148 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.205122 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e1c434b-dcac-4386-aaf2-4d8e3273de33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.205306 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.205159 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e1c434b-dcac-4386-aaf2-4d8e3273de33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.206815 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.206774 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e1c434b-dcac-4386-aaf2-4d8e3273de33-config-out\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.207144 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.207121 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.207215 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.207190 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.207680 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.207658 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.207730 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.207717 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.208207 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.208189 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-web-config\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.208279 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.208267 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.208702 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.208687 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e1c434b-dcac-4386-aaf2-4d8e3273de33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.208927 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.208906 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8e1c434b-dcac-4386-aaf2-4d8e3273de33-config-volume\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.218559 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.218522 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42797\" (UniqueName: \"kubernetes.io/projected/8e1c434b-dcac-4386-aaf2-4d8e3273de33-kube-api-access-42797\") pod \"alertmanager-main-0\" (UID: \"8e1c434b-dcac-4386-aaf2-4d8e3273de33\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.353947 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.353862 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 17:43:27.639224 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.639130 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 17:43:27.642031 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:43:27.642004 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1c434b_dcac_4386_aaf2_4d8e3273de33.slice/crio-8a193c19692fdf8b3d3ede2e51ecee4442fa6f9d6b4c0d8af4501aa09fcdb4fd WatchSource:0}: Error finding container 8a193c19692fdf8b3d3ede2e51ecee4442fa6f9d6b4c0d8af4501aa09fcdb4fd: Status 404 returned error can't find the container with id 8a193c19692fdf8b3d3ede2e51ecee4442fa6f9d6b4c0d8af4501aa09fcdb4fd Sep 17 17:43:27.995581 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.995545 2550 generic.go:356] "Generic (PLEG): container finished" podID="8e1c434b-dcac-4386-aaf2-4d8e3273de33" containerID="9f432159564383d7984e3b5c9058693ca190403790bac23f7d8b1c78ca293932" exitCode=0 Sep 17 17:43:27.995941 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.995592 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerDied","Data":"9f432159564383d7984e3b5c9058693ca190403790bac23f7d8b1c78ca293932"} Sep 17 17:43:27.995941 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:27.995663 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerStarted","Data":"8a193c19692fdf8b3d3ede2e51ecee4442fa6f9d6b4c0d8af4501aa09fcdb4fd"} Sep 17 17:43:28.214880 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:28.214857 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa" path="/var/lib/kubelet/pods/df249ff7-b4bc-4e8e-8e4c-6ba6a6e84efa/volumes" Sep 17 17:43:29.002901 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:29.002866 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerStarted","Data":"c3a997777a724dbac79902bd9f9e5c0b44a23cefc456b2c0e626b511d60d54ee"} Sep 17 17:43:29.003392 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:29.002929 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerStarted","Data":"5de71d10bce581d1942830ac3d37feea6098efdb8b08a0a473a9440d1bb0f645"} Sep 17 17:43:29.003392 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:29.002954 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerStarted","Data":"f60a584e77fa715734dcac3a43c5626b3aa9bcb3fc24c321e811a34c4c452bd8"} Sep 17 17:43:29.003392 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:29.002981 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerStarted","Data":"ccae7de3083d883437ae1c0c320c7a49960e99c23b0480c7bd46864479333d1f"} Sep 17 17:43:29.003392 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:29.003005 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerStarted","Data":"fccf4dd127cc536c94b44efc2383fe86b539a6d63076da10483a63472d8da63c"} Sep 17 17:43:29.003392 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:29.003029 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8e1c434b-dcac-4386-aaf2-4d8e3273de33","Type":"ContainerStarted","Data":"9fe5e520c1964117dc8747de612959ffa7283e0b4d71555733ca14727031d045"} Sep 17 17:43:29.033431 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:43:29.033368 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.014962524 podStartE2EDuration="2.033349835s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.387311ms" isStatefulPod=false podCreationTimestamp="2026-09-17 17:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:43:29.03149126 +0000 UTC m=+155.332058806" watchObservedRunningTime="2026-09-17 17:43:29.033349835 +0000 UTC m=+155.333917415" Sep 17 17:44:56.153811 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.153775 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq"] Sep 17 17:44:56.155791 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.155772 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.158635 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.158598 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 17 17:44:56.158778 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.158644 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 17 17:44:56.159457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.159438 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-w6w6b\"" Sep 17 17:44:56.166617 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.166594 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq"] Sep 17 17:44:56.261804 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.261770 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.261804 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.261808 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.262008 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.261844 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79p4z\" (UniqueName: \"kubernetes.io/projected/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-kube-api-access-79p4z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.362268 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.362231 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.362268 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.362270 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.362475 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.362298 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79p4z\" (UniqueName: \"kubernetes.io/projected/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-kube-api-access-79p4z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.362645 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.362628 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.362682 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.362669 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.370844 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.370817 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79p4z\" (UniqueName: \"kubernetes.io/projected/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-kube-api-access-79p4z\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.464591 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.464554 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:44:56.734581 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:56.734486 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq"] Sep 17 17:44:56.738015 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:44:56.737986 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1615f22_b7a0_47b1_b1cf_fe9cceda2f99.slice/crio-9f158dac051399180c7d1929e2be6aecba2530f212d3a9a6c0e11f5389ab49ed WatchSource:0}: Error finding container 9f158dac051399180c7d1929e2be6aecba2530f212d3a9a6c0e11f5389ab49ed: Status 404 returned error can't find the container with id 9f158dac051399180c7d1929e2be6aecba2530f212d3a9a6c0e11f5389ab49ed Sep 17 17:44:57.320836 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:44:57.320797 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" event={"ID":"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99","Type":"ContainerStarted","Data":"9f158dac051399180c7d1929e2be6aecba2530f212d3a9a6c0e11f5389ab49ed"} Sep 17 17:45:04.349579 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:04.349516 2550 generic.go:356] "Generic (PLEG): container finished" podID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerID="f5931e637910d5d6cfbeb97056d5ca64adf05fc4533cf4c13a955bf2a0c1dc10" exitCode=0 Sep 17 17:45:04.349965 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:04.349606 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" event={"ID":"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99","Type":"ContainerDied","Data":"f5931e637910d5d6cfbeb97056d5ca64adf05fc4533cf4c13a955bf2a0c1dc10"} Sep 17 17:45:06.359209 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:06.359175 2550 generic.go:356] "Generic (PLEG): container finished" podID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerID="0ce31ee73664c8dcb18181022ab2471213d10cc4b7a0450513ffea666bd3b66a" exitCode=0 Sep 17 17:45:06.359576 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:06.359248 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" event={"ID":"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99","Type":"ContainerDied","Data":"0ce31ee73664c8dcb18181022ab2471213d10cc4b7a0450513ffea666bd3b66a"} Sep 17 17:45:13.387730 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:13.387699 2550 generic.go:356] "Generic (PLEG): container finished" podID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerID="0851166a8cc056bf0481059754f1bbec1f4cf9e925c827d6d9ba5b2355815bf5" exitCode=0 Sep 17 17:45:13.388113 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:13.387764 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" event={"ID":"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99","Type":"ContainerDied","Data":"0851166a8cc056bf0481059754f1bbec1f4cf9e925c827d6d9ba5b2355815bf5"} Sep 17 17:45:14.516897 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.516871 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:45:14.608766 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.608729 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-bundle\") pod \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " Sep 17 17:45:14.608766 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.608766 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-util\") pod \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " Sep 17 17:45:14.609015 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.608790 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79p4z\" (UniqueName: \"kubernetes.io/projected/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-kube-api-access-79p4z\") pod \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\" (UID: \"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99\") " Sep 17 17:45:14.609449 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.609424 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-bundle" (OuterVolumeSpecName: "bundle") pod "a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" (UID: "a1615f22-b7a0-47b1-b1cf-fe9cceda2f99"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:45:14.610982 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.610954 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-kube-api-access-79p4z" (OuterVolumeSpecName: "kube-api-access-79p4z") pod "a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" (UID: "a1615f22-b7a0-47b1-b1cf-fe9cceda2f99"). InnerVolumeSpecName "kube-api-access-79p4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:45:14.613221 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.613197 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-util" (OuterVolumeSpecName: "util") pod "a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" (UID: "a1615f22-b7a0-47b1-b1cf-fe9cceda2f99"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 17:45:14.709682 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.709645 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-bundle\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:45:14.709682 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.709673 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-util\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:45:14.709682 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:14.709681 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79p4z\" (UniqueName: \"kubernetes.io/projected/a1615f22-b7a0-47b1-b1cf-fe9cceda2f99-kube-api-access-79p4z\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:45:15.396653 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:15.396620 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" Sep 17 17:45:15.396809 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:15.396620 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1afwrpq" event={"ID":"a1615f22-b7a0-47b1-b1cf-fe9cceda2f99","Type":"ContainerDied","Data":"9f158dac051399180c7d1929e2be6aecba2530f212d3a9a6c0e11f5389ab49ed"} Sep 17 17:45:15.396809 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:15.396748 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f158dac051399180c7d1929e2be6aecba2530f212d3a9a6c0e11f5389ab49ed" Sep 17 17:45:18.907403 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907330 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4"] Sep 17 17:45:18.907855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907592 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerName="pull" Sep 17 17:45:18.907855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907604 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerName="pull" Sep 17 17:45:18.907855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907622 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerName="util" Sep 17 17:45:18.907855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907627 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerName="util" Sep 17 17:45:18.907855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907636 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerName="extract" Sep 17 17:45:18.907855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907641 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerName="extract" Sep 17 17:45:18.907855 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.907683 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1615f22-b7a0-47b1-b1cf-fe9cceda2f99" containerName="extract" Sep 17 17:45:18.947825 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.947786 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4"] Sep 17 17:45:18.947981 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.947898 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:18.950588 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.950560 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 17 17:45:18.950588 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.950576 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 17 17:45:18.950789 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.950695 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 17 17:45:18.950789 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:18.950723 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-22z7c\"" Sep 17 17:45:19.043458 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.043428 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/126cb4e4-c012-4bd1-8c8c-4555d584222e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-khtx4\" (UID: \"126cb4e4-c012-4bd1-8c8c-4555d584222e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:19.043647 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.043486 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqvz\" (UniqueName: \"kubernetes.io/projected/126cb4e4-c012-4bd1-8c8c-4555d584222e-kube-api-access-vlqvz\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-khtx4\" (UID: \"126cb4e4-c012-4bd1-8c8c-4555d584222e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:19.144424 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.144383 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/126cb4e4-c012-4bd1-8c8c-4555d584222e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-khtx4\" (UID: \"126cb4e4-c012-4bd1-8c8c-4555d584222e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:19.144615 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.144464 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqvz\" (UniqueName: \"kubernetes.io/projected/126cb4e4-c012-4bd1-8c8c-4555d584222e-kube-api-access-vlqvz\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-khtx4\" (UID: \"126cb4e4-c012-4bd1-8c8c-4555d584222e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:19.146832 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.146812 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/126cb4e4-c012-4bd1-8c8c-4555d584222e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-khtx4\" (UID: \"126cb4e4-c012-4bd1-8c8c-4555d584222e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:19.153062 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.153041 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqvz\" (UniqueName: \"kubernetes.io/projected/126cb4e4-c012-4bd1-8c8c-4555d584222e-kube-api-access-vlqvz\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-khtx4\" (UID: \"126cb4e4-c012-4bd1-8c8c-4555d584222e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:19.257835 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.257745 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:19.554861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:19.554697 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4"] Sep 17 17:45:19.557748 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:45:19.557721 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126cb4e4_c012_4bd1_8c8c_4555d584222e.slice/crio-dd199208812832ec81df70cc3903569b551e50871dcf0ada1028d663137549be WatchSource:0}: Error finding container dd199208812832ec81df70cc3903569b551e50871dcf0ada1028d663137549be: Status 404 returned error can't find the container with id dd199208812832ec81df70cc3903569b551e50871dcf0ada1028d663137549be Sep 17 17:45:20.416450 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:20.416412 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" event={"ID":"126cb4e4-c012-4bd1-8c8c-4555d584222e","Type":"ContainerStarted","Data":"dd199208812832ec81df70cc3903569b551e50871dcf0ada1028d663137549be"} Sep 17 17:45:24.434348 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.434312 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" event={"ID":"126cb4e4-c012-4bd1-8c8c-4555d584222e","Type":"ContainerStarted","Data":"30663cea36dbc473cb3d44a3bdc997a66373cca9e8d01d0a09014f2097aed1dd"} Sep 17 17:45:24.434764 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.434454 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:24.458317 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.457925 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" podStartSLOduration=2.03714768 podStartE2EDuration="6.457906055s" totalImagesPullingTime="4.420758375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:24.455480959 +0000 UTC m=+270.756048513" watchObservedRunningTime="2026-09-17 17:45:24.457906055 +0000 UTC m=+270.758473598" Sep 17 17:45:24.746711 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.746645 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-9sh4d"] Sep 17 17:45:24.748770 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.748749 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:24.751209 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.751186 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 17 17:45:24.751316 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.751224 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-79lc7\"" Sep 17 17:45:24.751436 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.751421 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 17 17:45:24.761727 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.761703 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-9sh4d"] Sep 17 17:45:24.889845 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.889816 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vm6g\" (UniqueName: \"kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-kube-api-access-5vm6g\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:24.890001 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.889857 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/622210eb-120b-4070-9e0f-7db2bf6e98d7-cabundle0\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:24.890001 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.889923 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-certificates\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:24.990423 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.990386 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vm6g\" (UniqueName: \"kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-kube-api-access-5vm6g\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:24.990620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.990435 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/622210eb-120b-4070-9e0f-7db2bf6e98d7-cabundle0\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:24.990620 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.990483 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-certificates\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:24.990620 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:45:24.990598 2550 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Sep 17 17:45:24.990620 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:45:24.990617 2550 secret.go:281] references non-existent secret key: ca.crt Sep 17 17:45:24.990771 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:45:24.990627 2550 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 17 17:45:24.990771 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:45:24.990642 2550 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-9sh4d: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 17 17:45:24.990771 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:45:24.990709 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-certificates podName:622210eb-120b-4070-9e0f-7db2bf6e98d7 nodeName:}" failed. No retries permitted until 2026-09-17 17:45:25.490692196 +0000 UTC m=+271.791259717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-certificates") pod "keda-operator-579c8958f7-9sh4d" (UID: "622210eb-120b-4070-9e0f-7db2bf6e98d7") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 17 17:45:24.991130 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:24.991111 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/622210eb-120b-4070-9e0f-7db2bf6e98d7-cabundle0\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:25.001388 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.001333 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vm6g\" (UniqueName: \"kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-kube-api-access-5vm6g\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:25.103337 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.103300 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq"] Sep 17 17:45:25.105577 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.105557 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.107928 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.107904 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 17 17:45:25.116408 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.116381 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq"] Sep 17 17:45:25.292729 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.292649 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86kx\" (UniqueName: \"kubernetes.io/projected/3fa29357-f07d-4fc0-9d09-b490172d21dc-kube-api-access-d86kx\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.292870 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.292748 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3fa29357-f07d-4fc0-9d09-b490172d21dc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.292870 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.292800 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3fa29357-f07d-4fc0-9d09-b490172d21dc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.393497 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.393464 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3fa29357-f07d-4fc0-9d09-b490172d21dc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.393497 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.393503 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86kx\" (UniqueName: \"kubernetes.io/projected/3fa29357-f07d-4fc0-9d09-b490172d21dc-kube-api-access-d86kx\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.393718 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.393583 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3fa29357-f07d-4fc0-9d09-b490172d21dc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.393896 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.393876 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/3fa29357-f07d-4fc0-9d09-b490172d21dc-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.396024 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.395997 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3fa29357-f07d-4fc0-9d09-b490172d21dc-certificates\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.406472 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.406448 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86kx\" (UniqueName: \"kubernetes.io/projected/3fa29357-f07d-4fc0-9d09-b490172d21dc-kube-api-access-d86kx\") pod \"keda-metrics-apiserver-77857b7b98-xjtnq\" (UID: \"3fa29357-f07d-4fc0-9d09-b490172d21dc\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.415335 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.415315 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:25.494960 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.494929 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-certificates\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:25.497314 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.497276 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/622210eb-120b-4070-9e0f-7db2bf6e98d7-certificates\") pod \"keda-operator-579c8958f7-9sh4d\" (UID: \"622210eb-120b-4070-9e0f-7db2bf6e98d7\") " pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:25.577261 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.577188 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9hxbq"] Sep 17 17:45:25.579977 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.579961 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.583011 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.582993 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Sep 17 17:45:25.595293 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.595272 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9hxbq"] Sep 17 17:45:25.663710 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.663682 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:25.687098 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.687067 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq"] Sep 17 17:45:25.690744 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:45:25.690719 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa29357_f07d_4fc0_9d09_b490172d21dc.slice/crio-3f76f1008034fb169feea4b66e85af5027380727d28468038e238474abddddf0 WatchSource:0}: Error finding container 3f76f1008034fb169feea4b66e85af5027380727d28468038e238474abddddf0: Status 404 returned error can't find the container with id 3f76f1008034fb169feea4b66e85af5027380727d28468038e238474abddddf0 Sep 17 17:45:25.695923 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.695895 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/41921707-c39d-4070-b446-941941ce9e4b-certificates\") pod \"keda-admission-6c4d864b86-9hxbq\" (UID: \"41921707-c39d-4070-b446-941941ce9e4b\") " pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.696043 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.695964 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh969\" (UniqueName: \"kubernetes.io/projected/41921707-c39d-4070-b446-941941ce9e4b-kube-api-access-rh969\") pod \"keda-admission-6c4d864b86-9hxbq\" (UID: \"41921707-c39d-4070-b446-941941ce9e4b\") " pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.796863 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.796799 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/41921707-c39d-4070-b446-941941ce9e4b-certificates\") pod \"keda-admission-6c4d864b86-9hxbq\" (UID: \"41921707-c39d-4070-b446-941941ce9e4b\") " pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.797031 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.796883 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh969\" (UniqueName: \"kubernetes.io/projected/41921707-c39d-4070-b446-941941ce9e4b-kube-api-access-rh969\") pod \"keda-admission-6c4d864b86-9hxbq\" (UID: \"41921707-c39d-4070-b446-941941ce9e4b\") " pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.799209 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.799186 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/41921707-c39d-4070-b446-941941ce9e4b-certificates\") pod \"keda-admission-6c4d864b86-9hxbq\" (UID: \"41921707-c39d-4070-b446-941941ce9e4b\") " pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.805405 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.805387 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh969\" (UniqueName: \"kubernetes.io/projected/41921707-c39d-4070-b446-941941ce9e4b-kube-api-access-rh969\") pod \"keda-admission-6c4d864b86-9hxbq\" (UID: \"41921707-c39d-4070-b446-941941ce9e4b\") " pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.890037 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.889960 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:25.938040 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:25.937716 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-9sh4d"] Sep 17 17:45:25.939909 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:45:25.939880 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622210eb_120b_4070_9e0f_7db2bf6e98d7.slice/crio-48530eb85de80c631af4407ba7f394cf76958564fc6a84757bc82464092570bf WatchSource:0}: Error finding container 48530eb85de80c631af4407ba7f394cf76958564fc6a84757bc82464092570bf: Status 404 returned error can't find the container with id 48530eb85de80c631af4407ba7f394cf76958564fc6a84757bc82464092570bf Sep 17 17:45:26.174657 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:26.174621 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9hxbq"] Sep 17 17:45:26.178077 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:45:26.178047 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41921707_c39d_4070_b446_941941ce9e4b.slice/crio-fd25c91f2bbcaef706c738011abe62a0db00e6f929c48533f45f26def7634c1f WatchSource:0}: Error finding container fd25c91f2bbcaef706c738011abe62a0db00e6f929c48533f45f26def7634c1f: Status 404 returned error can't find the container with id fd25c91f2bbcaef706c738011abe62a0db00e6f929c48533f45f26def7634c1f Sep 17 17:45:26.443835 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:26.443753 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" event={"ID":"3fa29357-f07d-4fc0-9d09-b490172d21dc","Type":"ContainerStarted","Data":"3f76f1008034fb169feea4b66e85af5027380727d28468038e238474abddddf0"} Sep 17 17:45:26.445031 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:26.445003 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-9sh4d" event={"ID":"622210eb-120b-4070-9e0f-7db2bf6e98d7","Type":"ContainerStarted","Data":"48530eb85de80c631af4407ba7f394cf76958564fc6a84757bc82464092570bf"} Sep 17 17:45:26.446127 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:26.446103 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" event={"ID":"41921707-c39d-4070-b446-941941ce9e4b","Type":"ContainerStarted","Data":"fd25c91f2bbcaef706c738011abe62a0db00e6f929c48533f45f26def7634c1f"} Sep 17 17:45:28.461834 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:28.461798 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" event={"ID":"41921707-c39d-4070-b446-941941ce9e4b","Type":"ContainerStarted","Data":"bc392bdc57ec95f6d77fefe45586abaa4c61aee970948e8b7e12e676dcd0781a"} Sep 17 17:45:28.462302 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:28.461920 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:28.480160 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:28.480080 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" podStartSLOduration=1.748521362 podStartE2EDuration="3.48005863s" totalImagesPullingTime="1.731537268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:28.47939 +0000 UTC m=+274.779957544" watchObservedRunningTime="2026-09-17 17:45:28.48005863 +0000 UTC m=+274.780626172" Sep 17 17:45:30.471756 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:30.471720 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-9sh4d" event={"ID":"622210eb-120b-4070-9e0f-7db2bf6e98d7","Type":"ContainerStarted","Data":"8597bbf3fd492e6ef681bf17ad8f0eed75ac89f28cac20b9f65746edb16fda96"} Sep 17 17:45:30.472110 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:30.471854 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:30.490217 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:30.490171 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-9sh4d" podStartSLOduration=2.9090254250000003 podStartE2EDuration="6.490156176s" totalImagesPullingTime="3.581130751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:30.48914384 +0000 UTC m=+276.789711393" watchObservedRunningTime="2026-09-17 17:45:30.490156176 +0000 UTC m=+276.790723718" Sep 17 17:45:35.498329 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:35.498291 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" event={"ID":"3fa29357-f07d-4fc0-9d09-b490172d21dc","Type":"ContainerStarted","Data":"02a8ec82e3946bb48434cb6bd137577e770ad50e57bd15dba243e7059d52159e"} Sep 17 17:45:35.498754 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:35.498476 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:35.515401 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:35.515199 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" podStartSLOduration=1.077875139 podStartE2EDuration="10.515182783s" totalImagesPullingTime="9.437307644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:45:35.514895719 +0000 UTC m=+281.815463288" watchObservedRunningTime="2026-09-17 17:45:35.515182783 +0000 UTC m=+281.815750325" Sep 17 17:45:45.441345 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:45.441313 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-khtx4" Sep 17 17:45:46.507743 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:46.507718 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-xjtnq" Sep 17 17:45:49.468582 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:49.468546 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-9hxbq" Sep 17 17:45:51.479018 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:51.478990 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-9sh4d" Sep 17 17:45:54.106748 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:45:54.106717 2550 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 17:46:45.060902 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.060866 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-9mb7g"] Sep 17 17:46:45.064010 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.063991 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.066787 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.066756 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn"] Sep 17 17:46:45.067949 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.067918 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 17 17:46:45.068415 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.068400 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 17 17:46:45.068497 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.068408 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-nh88c\"" Sep 17 17:46:45.069796 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.069778 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.073043 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.073023 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kbl99\"" Sep 17 17:46:45.073119 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.073103 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Sep 17 17:46:45.085078 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.085057 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 17 17:46:45.105048 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.105014 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn"] Sep 17 17:46:45.106597 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.106570 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-9mb7g"] Sep 17 17:46:45.132412 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.132384 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d2722e-ee53-4ea5-9f98-dbd9772897d8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-sx7zn\" (UID: \"26d2722e-ee53-4ea5-9f98-dbd9772897d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.132611 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.132431 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5bdm\" (UniqueName: \"kubernetes.io/projected/26d2722e-ee53-4ea5-9f98-dbd9772897d8-kube-api-access-c5bdm\") pod \"llmisvc-controller-manager-645f9c8fd6-sx7zn\" (UID: \"26d2722e-ee53-4ea5-9f98-dbd9772897d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.132611 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.132469 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9lp\" (UniqueName: \"kubernetes.io/projected/4b4e68a1-01ba-4483-9391-5d0c914c23cd-kube-api-access-fg9lp\") pod \"kserve-controller-manager-5c5cdf4877-9mb7g\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.132611 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.132524 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b4e68a1-01ba-4483-9391-5d0c914c23cd-cert\") pod \"kserve-controller-manager-5c5cdf4877-9mb7g\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.233082 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.233043 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9lp\" (UniqueName: \"kubernetes.io/projected/4b4e68a1-01ba-4483-9391-5d0c914c23cd-kube-api-access-fg9lp\") pod \"kserve-controller-manager-5c5cdf4877-9mb7g\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.233286 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.233096 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b4e68a1-01ba-4483-9391-5d0c914c23cd-cert\") pod \"kserve-controller-manager-5c5cdf4877-9mb7g\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.233286 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.233160 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d2722e-ee53-4ea5-9f98-dbd9772897d8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-sx7zn\" (UID: \"26d2722e-ee53-4ea5-9f98-dbd9772897d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.233286 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.233197 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5bdm\" (UniqueName: \"kubernetes.io/projected/26d2722e-ee53-4ea5-9f98-dbd9772897d8-kube-api-access-c5bdm\") pod \"llmisvc-controller-manager-645f9c8fd6-sx7zn\" (UID: \"26d2722e-ee53-4ea5-9f98-dbd9772897d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.235628 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.235601 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b4e68a1-01ba-4483-9391-5d0c914c23cd-cert\") pod \"kserve-controller-manager-5c5cdf4877-9mb7g\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.235762 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.235737 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d2722e-ee53-4ea5-9f98-dbd9772897d8-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-sx7zn\" (UID: \"26d2722e-ee53-4ea5-9f98-dbd9772897d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.254374 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.254336 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5bdm\" (UniqueName: \"kubernetes.io/projected/26d2722e-ee53-4ea5-9f98-dbd9772897d8-kube-api-access-c5bdm\") pod \"llmisvc-controller-manager-645f9c8fd6-sx7zn\" (UID: \"26d2722e-ee53-4ea5-9f98-dbd9772897d8\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.254502 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.254383 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9lp\" (UniqueName: \"kubernetes.io/projected/4b4e68a1-01ba-4483-9391-5d0c914c23cd-kube-api-access-fg9lp\") pod \"kserve-controller-manager-5c5cdf4877-9mb7g\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.375199 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.375088 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:45.381278 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.381251 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:45.666470 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.666323 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn"] Sep 17 17:46:45.668428 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.668407 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-9mb7g"] Sep 17 17:46:45.669747 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:46:45.669724 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod26d2722e_ee53_4ea5_9f98_dbd9772897d8.slice/crio-0b26de4c36a575195eb3baf21296701f56ed9495b32e7298b92808051994cfaf WatchSource:0}: Error finding container 0b26de4c36a575195eb3baf21296701f56ed9495b32e7298b92808051994cfaf: Status 404 returned error can't find the container with id 0b26de4c36a575195eb3baf21296701f56ed9495b32e7298b92808051994cfaf Sep 17 17:46:45.671585 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.671567 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 17:46:45.671771 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:46:45.671752 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4e68a1_01ba_4483_9391_5d0c914c23cd.slice/crio-0b4cf86e1b234060c4b979c710591f9d1edb35fd4e699f36f817a3dfda0326a2 WatchSource:0}: Error finding container 0b4cf86e1b234060c4b979c710591f9d1edb35fd4e699f36f817a3dfda0326a2: Status 404 returned error can't find the container with id 0b4cf86e1b234060c4b979c710591f9d1edb35fd4e699f36f817a3dfda0326a2 Sep 17 17:46:45.800459 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.800426 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" event={"ID":"4b4e68a1-01ba-4483-9391-5d0c914c23cd","Type":"ContainerStarted","Data":"0b4cf86e1b234060c4b979c710591f9d1edb35fd4e699f36f817a3dfda0326a2"} Sep 17 17:46:45.801431 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:45.801407 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" event={"ID":"26d2722e-ee53-4ea5-9f98-dbd9772897d8","Type":"ContainerStarted","Data":"0b26de4c36a575195eb3baf21296701f56ed9495b32e7298b92808051994cfaf"} Sep 17 17:46:49.823847 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:49.823796 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" event={"ID":"26d2722e-ee53-4ea5-9f98-dbd9772897d8","Type":"ContainerStarted","Data":"61a2d4e6cd7d77a5d9887833d978a2766c721887dc6514895eb2011a96383f8d"} Sep 17 17:46:49.824304 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:49.823915 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:46:49.825301 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:49.825276 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" event={"ID":"4b4e68a1-01ba-4483-9391-5d0c914c23cd","Type":"ContainerStarted","Data":"a0a849b09e8aafe97105980ee44068a7685ac652f11e77661736b9ea8b45a3f4"} Sep 17 17:46:49.825425 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:49.825399 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:46:49.844790 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:46:49.844730 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" podStartSLOduration=1.658141207 podStartE2EDuration="4.844710823s" totalImagesPullingTime="3.186569616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:49.844602236 +0000 UTC m=+356.145169781" watchObservedRunningTime="2026-09-17 17:46:49.844710823 +0000 UTC m=+356.145278369" Sep 17 17:47:20.831923 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:20.831891 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-sx7zn" Sep 17 17:47:20.834679 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:20.834659 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:47:20.852284 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:20.852234 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" podStartSLOduration=33.664610396 podStartE2EDuration="36.852220593s" totalImagesPullingTime="3.187610197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:46:49.868064379 +0000 UTC m=+356.168631923" watchObservedRunningTime="2026-09-17 17:47:20.852220593 +0000 UTC m=+387.152788136" Sep 17 17:47:29.847869 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.847832 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-9mb7g"] Sep 17 17:47:29.848297 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.848067 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" podUID="4b4e68a1-01ba-4483-9391-5d0c914c23cd" containerName="manager" containerID="cri-o://a0a849b09e8aafe97105980ee44068a7685ac652f11e77661736b9ea8b45a3f4" gracePeriod=10 Sep 17 17:47:29.878866 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.878829 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-mmpxz"] Sep 17 17:47:29.880907 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.880890 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:29.894613 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.894583 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-mmpxz"] Sep 17 17:47:29.899373 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.899340 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4-cert\") pod \"kserve-controller-manager-5c5cdf4877-mmpxz\" (UID: \"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4\") " pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:29.899518 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.899498 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5lj\" (UniqueName: \"kubernetes.io/projected/4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4-kube-api-access-9c5lj\") pod \"kserve-controller-manager-5c5cdf4877-mmpxz\" (UID: \"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4\") " pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:29.999033 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.998997 2550 generic.go:356] "Generic (PLEG): container finished" podID="4b4e68a1-01ba-4483-9391-5d0c914c23cd" containerID="a0a849b09e8aafe97105980ee44068a7685ac652f11e77661736b9ea8b45a3f4" exitCode=0 Sep 17 17:47:29.999207 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:29.999053 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" event={"ID":"4b4e68a1-01ba-4483-9391-5d0c914c23cd","Type":"ContainerDied","Data":"a0a849b09e8aafe97105980ee44068a7685ac652f11e77661736b9ea8b45a3f4"} Sep 17 17:47:30.000294 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.000269 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5lj\" (UniqueName: \"kubernetes.io/projected/4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4-kube-api-access-9c5lj\") pod \"kserve-controller-manager-5c5cdf4877-mmpxz\" (UID: \"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4\") " pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:30.000418 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.000303 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4-cert\") pod \"kserve-controller-manager-5c5cdf4877-mmpxz\" (UID: \"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4\") " pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:30.002600 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.002578 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4-cert\") pod \"kserve-controller-manager-5c5cdf4877-mmpxz\" (UID: \"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4\") " pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:30.011503 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.011468 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5lj\" (UniqueName: \"kubernetes.io/projected/4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4-kube-api-access-9c5lj\") pod \"kserve-controller-manager-5c5cdf4877-mmpxz\" (UID: \"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4\") " pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:30.094003 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.093976 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:47:30.100777 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.100724 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b4e68a1-01ba-4483-9391-5d0c914c23cd-cert\") pod \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " Sep 17 17:47:30.100870 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.100779 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg9lp\" (UniqueName: \"kubernetes.io/projected/4b4e68a1-01ba-4483-9391-5d0c914c23cd-kube-api-access-fg9lp\") pod \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\" (UID: \"4b4e68a1-01ba-4483-9391-5d0c914c23cd\") " Sep 17 17:47:30.102978 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.102955 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4e68a1-01ba-4483-9391-5d0c914c23cd-cert" (OuterVolumeSpecName: "cert") pod "4b4e68a1-01ba-4483-9391-5d0c914c23cd" (UID: "4b4e68a1-01ba-4483-9391-5d0c914c23cd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:47:30.103085 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.102973 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4e68a1-01ba-4483-9391-5d0c914c23cd-kube-api-access-fg9lp" (OuterVolumeSpecName: "kube-api-access-fg9lp") pod "4b4e68a1-01ba-4483-9391-5d0c914c23cd" (UID: "4b4e68a1-01ba-4483-9391-5d0c914c23cd"). InnerVolumeSpecName "kube-api-access-fg9lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:47:30.201808 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.201765 2550 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b4e68a1-01ba-4483-9391-5d0c914c23cd-cert\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:47:30.201808 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.201803 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg9lp\" (UniqueName: \"kubernetes.io/projected/4b4e68a1-01ba-4483-9391-5d0c914c23cd-kube-api-access-fg9lp\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:47:30.234593 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.234554 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:30.503167 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:30.503129 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-mmpxz"] Sep 17 17:47:30.505962 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:47:30.505938 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8d9d7e_e4c5_4310_abc8_80c6dd6680d4.slice/crio-171715b8090bc3cd3bcc30c7aa0563e2132b41e131f13c21ed623d26634f2fb5 WatchSource:0}: Error finding container 171715b8090bc3cd3bcc30c7aa0563e2132b41e131f13c21ed623d26634f2fb5: Status 404 returned error can't find the container with id 171715b8090bc3cd3bcc30c7aa0563e2132b41e131f13c21ed623d26634f2fb5 Sep 17 17:47:31.006482 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.006453 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" Sep 17 17:47:31.006958 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.006453 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-9mb7g" event={"ID":"4b4e68a1-01ba-4483-9391-5d0c914c23cd","Type":"ContainerDied","Data":"0b4cf86e1b234060c4b979c710591f9d1edb35fd4e699f36f817a3dfda0326a2"} Sep 17 17:47:31.006958 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.006612 2550 scope.go:117] "RemoveContainer" containerID="a0a849b09e8aafe97105980ee44068a7685ac652f11e77661736b9ea8b45a3f4" Sep 17 17:47:31.008200 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.008178 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" event={"ID":"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4","Type":"ContainerStarted","Data":"2544e2da4af3d6d599d1a7532adff0e35b238078e6fc5cb8029fb5993be761e2"} Sep 17 17:47:31.008304 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.008224 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" event={"ID":"4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4","Type":"ContainerStarted","Data":"171715b8090bc3cd3bcc30c7aa0563e2132b41e131f13c21ed623d26634f2fb5"} Sep 17 17:47:31.008304 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.008261 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:47:31.024217 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.024187 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-9mb7g"] Sep 17 17:47:31.027657 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.027626 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5c5cdf4877-9mb7g"] Sep 17 17:47:31.047372 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:31.047319 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" podStartSLOduration=1.647782759 podStartE2EDuration="2.047298847s" totalImagesPullingTime="399.516088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:47:31.046309226 +0000 UTC m=+397.346876782" watchObservedRunningTime="2026-09-17 17:47:31.047298847 +0000 UTC m=+397.347866391" Sep 17 17:47:32.215808 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:47:32.215777 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4e68a1-01ba-4483-9391-5d0c914c23cd" path="/var/lib/kubelet/pods/4b4e68a1-01ba-4483-9391-5d0c914c23cd/volumes" Sep 17 17:48:02.018455 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:02.018422 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5c5cdf4877-mmpxz" Sep 17 17:48:03.641249 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.641210 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-4bjvs"] Sep 17 17:48:03.641670 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.641518 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4e68a1-01ba-4483-9391-5d0c914c23cd" containerName="manager" Sep 17 17:48:03.641670 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.641541 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4e68a1-01ba-4483-9391-5d0c914c23cd" containerName="manager" Sep 17 17:48:03.641670 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.641607 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4e68a1-01ba-4483-9391-5d0c914c23cd" containerName="manager" Sep 17 17:48:03.644151 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.644129 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.646722 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.646702 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Sep 17 17:48:03.646840 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.646806 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-2pnq8\"" Sep 17 17:48:03.647809 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.647791 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\"" Sep 17 17:48:03.671430 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.671400 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4bjvs"] Sep 17 17:48:03.754470 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.754434 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7skx\" (UniqueName: \"kubernetes.io/projected/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-kube-api-access-n7skx\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.754661 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.754483 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-metrics-certs\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.754661 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.754582 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-cert\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.855572 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.855508 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-cert\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.855745 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.855640 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7skx\" (UniqueName: \"kubernetes.io/projected/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-kube-api-access-n7skx\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.855745 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.855682 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-metrics-certs\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.858263 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.858237 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-metrics-certs\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.858396 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.858276 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-cert\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.870317 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.870278 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7skx\" (UniqueName: \"kubernetes.io/projected/8b079cd9-39b5-4cb2-adea-86276d5ab0b8-kube-api-access-n7skx\") pod \"odh-model-controller-7b966d7984-4bjvs\" (UID: \"8b079cd9-39b5-4cb2-adea-86276d5ab0b8\") " pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:03.954640 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:03.954597 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:04.245649 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:04.245560 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-4bjvs"] Sep 17 17:48:04.249468 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:48:04.249439 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b079cd9_39b5_4cb2_adea_86276d5ab0b8.slice/crio-88033aef60b5aec4bc811d760527903c57b1c3bf1b78fec98e7ee090c2b1bade WatchSource:0}: Error finding container 88033aef60b5aec4bc811d760527903c57b1c3bf1b78fec98e7ee090c2b1bade: Status 404 returned error can't find the container with id 88033aef60b5aec4bc811d760527903c57b1c3bf1b78fec98e7ee090c2b1bade Sep 17 17:48:05.166882 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:05.166842 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4bjvs" event={"ID":"8b079cd9-39b5-4cb2-adea-86276d5ab0b8","Type":"ContainerStarted","Data":"88033aef60b5aec4bc811d760527903c57b1c3bf1b78fec98e7ee090c2b1bade"} Sep 17 17:48:07.177589 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:07.177556 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-4bjvs" event={"ID":"8b079cd9-39b5-4cb2-adea-86276d5ab0b8","Type":"ContainerStarted","Data":"a618f3a3f2f53865c61f895e3679898c0bb46b2ffa308af5d0d34164de4cad1d"} Sep 17 17:48:07.178033 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:07.177717 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:07.197045 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:07.196994 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-4bjvs" podStartSLOduration=1.769981528 podStartE2EDuration="4.196975833s" totalImagesPullingTime="2.426994305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:48:07.195922501 +0000 UTC m=+433.496490045" watchObservedRunningTime="2026-09-17 17:48:07.196975833 +0000 UTC m=+433.497543406" Sep 17 17:48:18.184856 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:18.184779 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-4bjvs" Sep 17 17:48:54.121457 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:54.121421 2550 scope.go:117] "RemoveContainer" containerID="d9f59ae20a40115c80926a5a3fb4ecc136c40ed5a2120a527c8c2fffe5916d33" Sep 17 17:48:54.129199 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:54.129179 2550 scope.go:117] "RemoveContainer" containerID="0f14f62c58149bfc1429f1deb1deb03f21c4cf5518a2c0e5038b718ffd4c3682" Sep 17 17:48:54.136076 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:54.136056 2550 scope.go:117] "RemoveContainer" containerID="68c673ba9eac4b0bc56f03db9499d12bfa98d9307d33f1663d825b6312575e18" Sep 17 17:48:54.143258 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:54.143238 2550 scope.go:117] "RemoveContainer" containerID="fe44aa93ebf3d19101403e5fcc3816b8c9c9d18a79542b6674260df34c33db10" Sep 17 17:48:54.150400 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:54.150379 2550 scope.go:117] "RemoveContainer" containerID="a80dfc970b0cb7793ab5ba1ec6f74802fa18f19134d0a9e8c8c25177ec9f9739" Sep 17 17:48:54.158000 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:54.157978 2550 scope.go:117] "RemoveContainer" containerID="2a6a8dc54c5b00635e7fb82a4c7d76193bc10a7457f5786666c9941d1676bfb8" Sep 17 17:48:54.165523 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:48:54.165501 2550 scope.go:117] "RemoveContainer" containerID="ce7bf07b387f359512e7dddd949c94bfa70f5da9925106a25e7c2a9f17823f74" Sep 17 17:49:50.576145 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.576068 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7"] Sep 17 17:49:50.579165 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.579146 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.583962 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.583937 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 17 17:49:50.584081 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.583962 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d46ec-kube-rbac-proxy-sar-config\"" Sep 17 17:49:50.584953 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.584921 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9cm6p\"" Sep 17 17:49:50.585059 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.585043 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d46ec-predictor-serving-cert\"" Sep 17 17:49:50.585122 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.585066 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 17 17:49:50.595637 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.595612 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7"] Sep 17 17:49:50.631478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.631445 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjpkx\" (UniqueName: \"kubernetes.io/projected/ba34d9f1-3398-473e-8112-96a472348d6e-kube-api-access-hjpkx\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.631478 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.631479 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba34d9f1-3398-473e-8112-96a472348d6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.631716 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.631567 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba34d9f1-3398-473e-8112-96a472348d6e-proxy-tls\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.732857 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.732828 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba34d9f1-3398-473e-8112-96a472348d6e-proxy-tls\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.733043 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.732888 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjpkx\" (UniqueName: \"kubernetes.io/projected/ba34d9f1-3398-473e-8112-96a472348d6e-kube-api-access-hjpkx\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.733043 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.732916 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba34d9f1-3398-473e-8112-96a472348d6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.733622 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.733594 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba34d9f1-3398-473e-8112-96a472348d6e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.735201 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.735173 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba34d9f1-3398-473e-8112-96a472348d6e-proxy-tls\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.742109 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.742084 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjpkx\" (UniqueName: \"kubernetes.io/projected/ba34d9f1-3398-473e-8112-96a472348d6e-kube-api-access-hjpkx\") pod \"success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:50.889471 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:50.889386 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:49:51.162866 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:51.161949 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7"] Sep 17 17:49:51.168990 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:49:51.168958 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba34d9f1_3398_473e_8112_96a472348d6e.slice/crio-ada50f61c734557a3c916623e909f03bfac699abc52d417cd49f9924f3af3767 WatchSource:0}: Error finding container ada50f61c734557a3c916623e909f03bfac699abc52d417cd49f9924f3af3767: Status 404 returned error can't find the container with id ada50f61c734557a3c916623e909f03bfac699abc52d417cd49f9924f3af3767 Sep 17 17:49:51.626842 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:49:51.626767 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" event={"ID":"ba34d9f1-3398-473e-8112-96a472348d6e","Type":"ContainerStarted","Data":"ada50f61c734557a3c916623e909f03bfac699abc52d417cd49f9924f3af3767"} Sep 17 17:50:03.693146 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:03.693102 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" event={"ID":"ba34d9f1-3398-473e-8112-96a472348d6e","Type":"ContainerStarted","Data":"8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf"} Sep 17 17:50:05.705118 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:05.705019 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" event={"ID":"ba34d9f1-3398-473e-8112-96a472348d6e","Type":"ContainerStarted","Data":"7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7"} Sep 17 17:50:05.705582 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:05.705263 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:50:05.705582 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:05.705310 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:50:05.706764 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:05.706731 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 17 17:50:05.742922 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:05.742853 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podStartSLOduration=1.5608308260000001 podStartE2EDuration="15.742831923s" totalImagesPullingTime="14.182001097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:49:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:50:05.742828374 +0000 UTC m=+552.043395914" watchObservedRunningTime="2026-09-17 17:50:05.742831923 +0000 UTC m=+552.043399467" Sep 17 17:50:06.709543 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:06.709498 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 17 17:50:11.714485 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:11.714458 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:50:11.714995 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:11.714965 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 17 17:50:21.715544 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:21.715499 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 17 17:50:31.715351 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:31.715307 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 17 17:50:41.715539 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:41.715500 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 17 17:50:51.715806 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:50:51.715776 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:51:10.357643 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.357614 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4"] Sep 17 17:51:10.360319 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.360303 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:10.363991 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.363970 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d46ec-serving-cert\"" Sep 17 17:51:10.364102 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.363970 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-d46ec-kube-rbac-proxy-sar-config\"" Sep 17 17:51:10.369996 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.369973 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4"] Sep 17 17:51:10.408515 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.408480 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls\") pod \"switch-graph-d46ec-7f8c97c56d-rqgm4\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:10.408515 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.408517 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09dca777-3158-4cbd-b43c-f75074395091-openshift-service-ca-bundle\") pod \"switch-graph-d46ec-7f8c97c56d-rqgm4\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:10.509169 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.509132 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls\") pod \"switch-graph-d46ec-7f8c97c56d-rqgm4\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:10.509337 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.509185 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09dca777-3158-4cbd-b43c-f75074395091-openshift-service-ca-bundle\") pod \"switch-graph-d46ec-7f8c97c56d-rqgm4\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:10.509337 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:51:10.509286 2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-d46ec-serving-cert: secret "switch-graph-d46ec-serving-cert" not found Sep 17 17:51:10.509410 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:51:10.509393 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls podName:09dca777-3158-4cbd-b43c-f75074395091 nodeName:}" failed. No retries permitted until 2026-09-17 17:51:11.009371645 +0000 UTC m=+617.309939171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls") pod "switch-graph-d46ec-7f8c97c56d-rqgm4" (UID: "09dca777-3158-4cbd-b43c-f75074395091") : secret "switch-graph-d46ec-serving-cert" not found Sep 17 17:51:10.509900 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:10.509881 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09dca777-3158-4cbd-b43c-f75074395091-openshift-service-ca-bundle\") pod \"switch-graph-d46ec-7f8c97c56d-rqgm4\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:11.014971 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:11.014925 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls\") pod \"switch-graph-d46ec-7f8c97c56d-rqgm4\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:11.017387 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:11.017368 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls\") pod \"switch-graph-d46ec-7f8c97c56d-rqgm4\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:11.270905 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:11.270822 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:11.542269 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:11.542244 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4"] Sep 17 17:51:11.544263 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:51:11.544236 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dca777_3158_4cbd_b43c_f75074395091.slice/crio-9e6b31baa8defcd23232ee4aa719c5f2bce4f4dce7e3e2937a7b9034bbe8a6a0 WatchSource:0}: Error finding container 9e6b31baa8defcd23232ee4aa719c5f2bce4f4dce7e3e2937a7b9034bbe8a6a0: Status 404 returned error can't find the container with id 9e6b31baa8defcd23232ee4aa719c5f2bce4f4dce7e3e2937a7b9034bbe8a6a0 Sep 17 17:51:11.985723 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:11.985693 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" event={"ID":"09dca777-3158-4cbd-b43c-f75074395091","Type":"ContainerStarted","Data":"9e6b31baa8defcd23232ee4aa719c5f2bce4f4dce7e3e2937a7b9034bbe8a6a0"} Sep 17 17:51:13.996199 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:13.996167 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" event={"ID":"09dca777-3158-4cbd-b43c-f75074395091","Type":"ContainerStarted","Data":"d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13"} Sep 17 17:51:13.996607 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:13.996279 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:14.014132 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:14.014084 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podStartSLOduration=2.00911209 podStartE2EDuration="4.014067788s" totalImagesPullingTime="2.004955698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:14.011967794 +0000 UTC m=+620.312535348" watchObservedRunningTime="2026-09-17 17:51:14.014067788 +0000 UTC m=+620.314635330" Sep 17 17:51:20.006999 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.006929 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:20.557102 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.557070 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4"] Sep 17 17:51:20.557270 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.557246 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" containerID="cri-o://d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13" gracePeriod=30 Sep 17 17:51:20.641201 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.641168 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7"] Sep 17 17:51:20.641517 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.641466 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" containerID="cri-o://8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf" gracePeriod=30 Sep 17 17:51:20.641675 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.641516 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kube-rbac-proxy" containerID="cri-o://7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7" gracePeriod=30 Sep 17 17:51:20.763365 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.763338 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm"] Sep 17 17:51:20.766670 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.766650 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:20.769386 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.769356 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8592d-kube-rbac-proxy-sar-config\"" Sep 17 17:51:20.769497 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.769400 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8592d-predictor-serving-cert\"" Sep 17 17:51:20.782504 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.782478 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm"] Sep 17 17:51:20.901293 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.901215 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-proxy-tls\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:20.901293 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.901282 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8ln\" (UniqueName: \"kubernetes.io/projected/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-api-access-rp8ln\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:20.901474 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:20.901323 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.002072 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.002039 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-proxy-tls\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.002237 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.002106 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8ln\" (UniqueName: \"kubernetes.io/projected/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-api-access-rp8ln\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.002237 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.002150 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.002845 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.002821 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.004418 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.004399 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-proxy-tls\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.012499 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.012478 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8ln\" (UniqueName: \"kubernetes.io/projected/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-api-access-rp8ln\") pod \"success-200-isvc-8592d-predictor-b4d845d84-vrdmm\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.025869 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.025836 2550 generic.go:356] "Generic (PLEG): container finished" podID="ba34d9f1-3398-473e-8112-96a472348d6e" containerID="7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7" exitCode=2 Sep 17 17:51:21.026004 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.025905 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" event={"ID":"ba34d9f1-3398-473e-8112-96a472348d6e","Type":"ContainerDied","Data":"7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7"} Sep 17 17:51:21.076778 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.076739 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:21.352025 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.352002 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm"] Sep 17 17:51:21.354304 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:51:21.354276 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ee9c65_b9ff_4d87_ab12_60bc80b5d9c4.slice/crio-ec3cca6fa64414874036bbf684c851261256a47b5cc3bd2566fc9398bbb7a529 WatchSource:0}: Error finding container ec3cca6fa64414874036bbf684c851261256a47b5cc3bd2566fc9398bbb7a529: Status 404 returned error can't find the container with id ec3cca6fa64414874036bbf684c851261256a47b5cc3bd2566fc9398bbb7a529 Sep 17 17:51:21.709922 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.709883 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused" Sep 17 17:51:21.715294 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:21.715260 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused" Sep 17 17:51:22.031480 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:22.031393 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" event={"ID":"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4","Type":"ContainerStarted","Data":"a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce"} Sep 17 17:51:22.031480 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:22.031457 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:22.031480 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:22.031478 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" event={"ID":"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4","Type":"ContainerStarted","Data":"510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04"} Sep 17 17:51:22.032041 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:22.031505 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" event={"ID":"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4","Type":"ContainerStarted","Data":"ec3cca6fa64414874036bbf684c851261256a47b5cc3bd2566fc9398bbb7a529"} Sep 17 17:51:22.032041 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:22.031547 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:22.033009 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:22.032983 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 17 17:51:22.070483 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:22.070429 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podStartSLOduration=2.070413135 podStartE2EDuration="2.070413135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:51:22.068644968 +0000 UTC m=+628.369212512" watchObservedRunningTime="2026-09-17 17:51:22.070413135 +0000 UTC m=+628.370980677" Sep 17 17:51:23.035975 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.035937 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 17 17:51:23.686237 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.686205 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:51:23.824780 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.824684 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba34d9f1-3398-473e-8112-96a472348d6e-kube-rbac-proxy-sar-config\") pod \"ba34d9f1-3398-473e-8112-96a472348d6e\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " Sep 17 17:51:23.824780 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.824728 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjpkx\" (UniqueName: \"kubernetes.io/projected/ba34d9f1-3398-473e-8112-96a472348d6e-kube-api-access-hjpkx\") pod \"ba34d9f1-3398-473e-8112-96a472348d6e\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " Sep 17 17:51:23.824780 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.824784 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba34d9f1-3398-473e-8112-96a472348d6e-proxy-tls\") pod \"ba34d9f1-3398-473e-8112-96a472348d6e\" (UID: \"ba34d9f1-3398-473e-8112-96a472348d6e\") " Sep 17 17:51:23.825143 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.825118 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba34d9f1-3398-473e-8112-96a472348d6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba34d9f1-3398-473e-8112-96a472348d6e" (UID: "ba34d9f1-3398-473e-8112-96a472348d6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:51:23.827026 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.827004 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba34d9f1-3398-473e-8112-96a472348d6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba34d9f1-3398-473e-8112-96a472348d6e" (UID: "ba34d9f1-3398-473e-8112-96a472348d6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:51:23.827077 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.827024 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba34d9f1-3398-473e-8112-96a472348d6e-kube-api-access-hjpkx" (OuterVolumeSpecName: "kube-api-access-hjpkx") pod "ba34d9f1-3398-473e-8112-96a472348d6e" (UID: "ba34d9f1-3398-473e-8112-96a472348d6e"). InnerVolumeSpecName "kube-api-access-hjpkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 17:51:23.925689 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.925653 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba34d9f1-3398-473e-8112-96a472348d6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:51:23.925689 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.925683 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjpkx\" (UniqueName: \"kubernetes.io/projected/ba34d9f1-3398-473e-8112-96a472348d6e-kube-api-access-hjpkx\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:51:23.925689 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:23.925696 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba34d9f1-3398-473e-8112-96a472348d6e-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:51:24.042021 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.041985 2550 generic.go:356] "Generic (PLEG): container finished" podID="ba34d9f1-3398-473e-8112-96a472348d6e" containerID="8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf" exitCode=0 Sep 17 17:51:24.042410 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.042033 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" event={"ID":"ba34d9f1-3398-473e-8112-96a472348d6e","Type":"ContainerDied","Data":"8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf"} Sep 17 17:51:24.042410 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.042038 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" Sep 17 17:51:24.042410 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.042093 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7" event={"ID":"ba34d9f1-3398-473e-8112-96a472348d6e","Type":"ContainerDied","Data":"ada50f61c734557a3c916623e909f03bfac699abc52d417cd49f9924f3af3767"} Sep 17 17:51:24.042410 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.042129 2550 scope.go:117] "RemoveContainer" containerID="7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7" Sep 17 17:51:24.050638 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.050619 2550 scope.go:117] "RemoveContainer" containerID="8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf" Sep 17 17:51:24.057594 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.057577 2550 scope.go:117] "RemoveContainer" containerID="7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7" Sep 17 17:51:24.057917 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:51:24.057885 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7\": container with ID starting with 7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7 not found: ID does not exist" containerID="7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7" Sep 17 17:51:24.058014 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.057924 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7"} err="failed to get container status \"7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7\": rpc error: code = NotFound desc = could not find container \"7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7\": container with ID starting with 7b682a83f3b4835091c288e91226dd12d6dd117b0a3ebe4e909393f2998291e7 not found: ID does not exist" Sep 17 17:51:24.058014 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.057962 2550 scope.go:117] "RemoveContainer" containerID="8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf" Sep 17 17:51:24.058227 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:51:24.058210 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf\": container with ID starting with 8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf not found: ID does not exist" containerID="8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf" Sep 17 17:51:24.058280 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.058233 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf"} err="failed to get container status \"8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf\": rpc error: code = NotFound desc = could not find container \"8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf\": container with ID starting with 8a5a0e3034be9621c52fc83c4ca041bdc5ff91150125f03bce504403da395dcf not found: ID does not exist" Sep 17 17:51:24.063429 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.063405 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7"] Sep 17 17:51:24.068558 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.068539 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d46ec-predictor-54ff8f5568-kj6n7"] Sep 17 17:51:24.216200 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:24.216163 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" path="/var/lib/kubelet/pods/ba34d9f1-3398-473e-8112-96a472348d6e/volumes" Sep 17 17:51:25.004512 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:25.004469 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:51:28.041885 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:28.041856 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:51:28.042406 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:28.042382 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 17 17:51:30.005220 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:30.005182 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:51:35.005228 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:35.005194 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:51:35.007262 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:35.005297 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:38.042845 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:38.042798 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 17 17:51:40.005272 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:40.005236 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:51:45.004443 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:45.004395 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:51:48.043277 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:48.043222 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 17 17:51:50.010250 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.010207 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:51:50.710389 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.710364 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:50.727861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.727660 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls\") pod \"09dca777-3158-4cbd-b43c-f75074395091\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " Sep 17 17:51:50.727861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.727728 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09dca777-3158-4cbd-b43c-f75074395091-openshift-service-ca-bundle\") pod \"09dca777-3158-4cbd-b43c-f75074395091\" (UID: \"09dca777-3158-4cbd-b43c-f75074395091\") " Sep 17 17:51:50.728271 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.728244 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dca777-3158-4cbd-b43c-f75074395091-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "09dca777-3158-4cbd-b43c-f75074395091" (UID: "09dca777-3158-4cbd-b43c-f75074395091"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:51:50.730002 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.729969 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09dca777-3158-4cbd-b43c-f75074395091" (UID: "09dca777-3158-4cbd-b43c-f75074395091"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:51:50.828592 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.828488 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09dca777-3158-4cbd-b43c-f75074395091-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:51:50.828592 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:50.828520 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09dca777-3158-4cbd-b43c-f75074395091-openshift-service-ca-bundle\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:51:51.165253 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.165217 2550 generic.go:356] "Generic (PLEG): container finished" podID="09dca777-3158-4cbd-b43c-f75074395091" containerID="d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13" exitCode=0 Sep 17 17:51:51.165709 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.165260 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" Sep 17 17:51:51.165709 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.165268 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" event={"ID":"09dca777-3158-4cbd-b43c-f75074395091","Type":"ContainerDied","Data":"d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13"} Sep 17 17:51:51.165709 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.165319 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4" event={"ID":"09dca777-3158-4cbd-b43c-f75074395091","Type":"ContainerDied","Data":"9e6b31baa8defcd23232ee4aa719c5f2bce4f4dce7e3e2937a7b9034bbe8a6a0"} Sep 17 17:51:51.165709 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.165352 2550 scope.go:117] "RemoveContainer" containerID="d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13" Sep 17 17:51:51.173878 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.173857 2550 scope.go:117] "RemoveContainer" containerID="d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13" Sep 17 17:51:51.174211 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:51:51.174192 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13\": container with ID starting with d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13 not found: ID does not exist" containerID="d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13" Sep 17 17:51:51.174304 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.174217 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13"} err="failed to get container status \"d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13\": rpc error: code = NotFound desc = could not find container \"d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13\": container with ID starting with d93e0994215dcfec0da11487b2e829c8d9b2b9dca2a32ce6946e2a738e365b13 not found: ID does not exist" Sep 17 17:51:51.186732 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.186707 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4"] Sep 17 17:51:51.190647 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:51.190624 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-d46ec-7f8c97c56d-rqgm4"] Sep 17 17:51:52.214957 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:52.214916 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09dca777-3158-4cbd-b43c-f75074395091" path="/var/lib/kubelet/pods/09dca777-3158-4cbd-b43c-f75074395091/volumes" Sep 17 17:51:58.043020 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:51:58.042980 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 17 17:52:00.370981 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.370944 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s"] Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371287 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371299 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371312 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kube-rbac-proxy" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371319 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kube-rbac-proxy" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371325 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371330 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371375 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="09dca777-3158-4cbd-b43c-f75074395091" containerName="switch-graph-d46ec" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371383 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kserve-container" Sep 17 17:52:00.371409 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.371390 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba34d9f1-3398-473e-8112-96a472348d6e" containerName="kube-rbac-proxy" Sep 17 17:52:00.375759 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.375740 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:00.378583 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.378560 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Sep 17 17:52:00.378690 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.378561 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Sep 17 17:52:00.385510 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.385486 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s"] Sep 17 17:52:00.403913 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.403875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/749b19de-b26d-4b84-97c3-a639fda6e829-openshift-service-ca-bundle\") pod \"model-chainer-76bc4cf9d7-2h22s\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:00.403913 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.403918 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls\") pod \"model-chainer-76bc4cf9d7-2h22s\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:00.504850 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.504814 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/749b19de-b26d-4b84-97c3-a639fda6e829-openshift-service-ca-bundle\") pod \"model-chainer-76bc4cf9d7-2h22s\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:00.504850 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.504853 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls\") pod \"model-chainer-76bc4cf9d7-2h22s\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:00.505046 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:52:00.504965 2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Sep 17 17:52:00.505046 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:52:00.505032 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls podName:749b19de-b26d-4b84-97c3-a639fda6e829 nodeName:}" failed. No retries permitted until 2026-09-17 17:52:01.005011562 +0000 UTC m=+667.305579084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls") pod "model-chainer-76bc4cf9d7-2h22s" (UID: "749b19de-b26d-4b84-97c3-a639fda6e829") : secret "model-chainer-serving-cert" not found Sep 17 17:52:00.505491 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:00.505471 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/749b19de-b26d-4b84-97c3-a639fda6e829-openshift-service-ca-bundle\") pod \"model-chainer-76bc4cf9d7-2h22s\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:01.009575 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:01.009518 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls\") pod \"model-chainer-76bc4cf9d7-2h22s\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:01.011997 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:01.011974 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls\") pod \"model-chainer-76bc4cf9d7-2h22s\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:01.285930 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:01.285843 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:01.555861 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:01.555781 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s"] Sep 17 17:52:01.558772 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:52:01.558737 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749b19de_b26d_4b84_97c3_a639fda6e829.slice/crio-351840f114fec588e7aaa7ca406ea74bbf62047181e84ea9219af003965416e7 WatchSource:0}: Error finding container 351840f114fec588e7aaa7ca406ea74bbf62047181e84ea9219af003965416e7: Status 404 returned error can't find the container with id 351840f114fec588e7aaa7ca406ea74bbf62047181e84ea9219af003965416e7 Sep 17 17:52:02.215114 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:02.215074 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" event={"ID":"749b19de-b26d-4b84-97c3-a639fda6e829","Type":"ContainerStarted","Data":"eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df"} Sep 17 17:52:02.215114 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:02.215121 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:02.215343 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:02.215135 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" event={"ID":"749b19de-b26d-4b84-97c3-a639fda6e829","Type":"ContainerStarted","Data":"351840f114fec588e7aaa7ca406ea74bbf62047181e84ea9219af003965416e7"} Sep 17 17:52:02.234095 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:02.234039 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podStartSLOduration=2.234022024 podStartE2EDuration="2.234022024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:02.232115324 +0000 UTC m=+668.532682870" watchObservedRunningTime="2026-09-17 17:52:02.234022024 +0000 UTC m=+668.534589621" Sep 17 17:52:08.043811 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:08.043782 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 17:52:09.229140 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:09.229110 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:10.945363 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:10.945330 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s"] Sep 17 17:52:10.945821 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:10.945569 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" containerID="cri-o://eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df" gracePeriod=30 Sep 17 17:52:11.233610 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.233508 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp"] Sep 17 17:52:11.237017 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.237001 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.239303 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.239280 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-081d8-predictor-serving-cert\"" Sep 17 17:52:11.239587 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.239564 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-081d8-kube-rbac-proxy-sar-config\"" Sep 17 17:52:11.248420 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.248396 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp"] Sep 17 17:52:11.290269 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.290232 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.290442 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.290276 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zml7\" (UniqueName: \"kubernetes.io/projected/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-api-access-5zml7\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.290442 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.290314 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df9a881-bfd4-4352-8f03-d0060649fb9d-proxy-tls\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.391793 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.391754 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.391793 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.391793 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zml7\" (UniqueName: \"kubernetes.io/projected/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-api-access-5zml7\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.392041 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.391818 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df9a881-bfd4-4352-8f03-d0060649fb9d-proxy-tls\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.392450 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.392425 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.394173 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.394153 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df9a881-bfd4-4352-8f03-d0060649fb9d-proxy-tls\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.401864 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.401839 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zml7\" (UniqueName: \"kubernetes.io/projected/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-api-access-5zml7\") pod \"success-200-isvc-081d8-predictor-6b56b68db-qlfsp\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.548596 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.548489 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:11.823726 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:11.823649 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp"] Sep 17 17:52:11.826791 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:52:11.826757 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df9a881_bfd4_4352_8f03_d0060649fb9d.slice/crio-8daebeaa1be5e0f5ce4a51ddf5215b343bc7c1e9f9fe9f268f3e067d53b33bc3 WatchSource:0}: Error finding container 8daebeaa1be5e0f5ce4a51ddf5215b343bc7c1e9f9fe9f268f3e067d53b33bc3: Status 404 returned error can't find the container with id 8daebeaa1be5e0f5ce4a51ddf5215b343bc7c1e9f9fe9f268f3e067d53b33bc3 Sep 17 17:52:12.258439 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:12.258407 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" event={"ID":"9df9a881-bfd4-4352-8f03-d0060649fb9d","Type":"ContainerStarted","Data":"2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296"} Sep 17 17:52:12.258924 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:12.258468 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:12.258924 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:12.258494 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:12.258924 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:12.258511 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" event={"ID":"9df9a881-bfd4-4352-8f03-d0060649fb9d","Type":"ContainerStarted","Data":"6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541"} Sep 17 17:52:12.258924 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:12.258544 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" event={"ID":"9df9a881-bfd4-4352-8f03-d0060649fb9d","Type":"ContainerStarted","Data":"8daebeaa1be5e0f5ce4a51ddf5215b343bc7c1e9f9fe9f268f3e067d53b33bc3"} Sep 17 17:52:12.260141 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:12.260111 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 17 17:52:12.278061 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:12.278005 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podStartSLOduration=1.277988782 podStartE2EDuration="1.277988782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:12.275925796 +0000 UTC m=+678.576493341" watchObservedRunningTime="2026-09-17 17:52:12.277988782 +0000 UTC m=+678.578556327" Sep 17 17:52:13.262398 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:13.262362 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 17 17:52:14.228978 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:14.228938 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:52:18.267574 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:18.267546 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 17:52:18.268167 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:18.267958 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 17 17:52:19.228210 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:19.228169 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:52:20.807419 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.807379 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9"] Sep 17 17:52:20.810809 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.810788 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:20.813639 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.813610 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8592d-serving-cert\"" Sep 17 17:52:20.813799 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.813612 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-8592d-kube-rbac-proxy-sar-config\"" Sep 17 17:52:20.823904 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.823872 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9"] Sep 17 17:52:20.873062 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.873020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-openshift-service-ca-bundle\") pod \"switch-graph-8592d-688fc594f9-lcxv9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:20.873227 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.873073 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-proxy-tls\") pod \"switch-graph-8592d-688fc594f9-lcxv9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:20.974147 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.974104 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-openshift-service-ca-bundle\") pod \"switch-graph-8592d-688fc594f9-lcxv9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:20.974147 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.974154 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-proxy-tls\") pod \"switch-graph-8592d-688fc594f9-lcxv9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:20.974813 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.974786 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-openshift-service-ca-bundle\") pod \"switch-graph-8592d-688fc594f9-lcxv9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:20.976601 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:20.976575 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-proxy-tls\") pod \"switch-graph-8592d-688fc594f9-lcxv9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:21.122950 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:21.122855 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:21.411386 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:21.411340 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9"] Sep 17 17:52:21.415454 ip-10-0-135-40 kubenswrapper[2550]: W0917 17:52:21.415427 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01585e3c_a3a1_4b0d_b15d_52eedb9ccaa9.slice/crio-b0ab305e766842116edb4f4ffe49b28e49b2a8aa60236e5d64a31a891a23f144 WatchSource:0}: Error finding container b0ab305e766842116edb4f4ffe49b28e49b2a8aa60236e5d64a31a891a23f144: Status 404 returned error can't find the container with id b0ab305e766842116edb4f4ffe49b28e49b2a8aa60236e5d64a31a891a23f144 Sep 17 17:52:22.303204 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:22.303156 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" event={"ID":"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9","Type":"ContainerStarted","Data":"0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d"} Sep 17 17:52:22.303670 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:22.303222 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" event={"ID":"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9","Type":"ContainerStarted","Data":"b0ab305e766842116edb4f4ffe49b28e49b2a8aa60236e5d64a31a891a23f144"} Sep 17 17:52:22.303670 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:22.303271 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:22.320418 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:22.320366 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podStartSLOduration=2.320347254 podStartE2EDuration="2.320347254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 17:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 17:52:22.318899032 +0000 UTC m=+688.619466577" watchObservedRunningTime="2026-09-17 17:52:22.320347254 +0000 UTC m=+688.620914797" Sep 17 17:52:24.227578 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:24.227540 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:52:24.228041 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:24.227635 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:28.268864 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:28.268826 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 17 17:52:28.314245 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:28.314211 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 17:52:29.227877 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:29.227840 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:52:34.227442 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:34.227407 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:52:38.268253 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:38.268210 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 17 17:52:39.228153 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:39.228115 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 17:52:41.096267 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.096242 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:41.248722 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.248620 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls\") pod \"749b19de-b26d-4b84-97c3-a639fda6e829\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " Sep 17 17:52:41.248885 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.248748 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/749b19de-b26d-4b84-97c3-a639fda6e829-openshift-service-ca-bundle\") pod \"749b19de-b26d-4b84-97c3-a639fda6e829\" (UID: \"749b19de-b26d-4b84-97c3-a639fda6e829\") " Sep 17 17:52:41.249122 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.249096 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749b19de-b26d-4b84-97c3-a639fda6e829-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "749b19de-b26d-4b84-97c3-a639fda6e829" (UID: "749b19de-b26d-4b84-97c3-a639fda6e829"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 17:52:41.250718 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.250695 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "749b19de-b26d-4b84-97c3-a639fda6e829" (UID: "749b19de-b26d-4b84-97c3-a639fda6e829"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 17:52:41.349493 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.349457 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/749b19de-b26d-4b84-97c3-a639fda6e829-openshift-service-ca-bundle\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:52:41.349493 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.349487 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/749b19de-b26d-4b84-97c3-a639fda6e829-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 17:52:41.395491 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.395443 2550 generic.go:356] "Generic (PLEG): container finished" podID="749b19de-b26d-4b84-97c3-a639fda6e829" containerID="eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df" exitCode=0 Sep 17 17:52:41.395491 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.395491 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" Sep 17 17:52:41.395708 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.395553 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" event={"ID":"749b19de-b26d-4b84-97c3-a639fda6e829","Type":"ContainerDied","Data":"eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df"} Sep 17 17:52:41.395708 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.395614 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s" event={"ID":"749b19de-b26d-4b84-97c3-a639fda6e829","Type":"ContainerDied","Data":"351840f114fec588e7aaa7ca406ea74bbf62047181e84ea9219af003965416e7"} Sep 17 17:52:41.395708 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.395637 2550 scope.go:117] "RemoveContainer" containerID="eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df" Sep 17 17:52:41.404103 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.404079 2550 scope.go:117] "RemoveContainer" containerID="eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df" Sep 17 17:52:41.404430 ip-10-0-135-40 kubenswrapper[2550]: E0917 17:52:41.404409 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df\": container with ID starting with eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df not found: ID does not exist" containerID="eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df" Sep 17 17:52:41.404511 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.404439 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df"} err="failed to get container status \"eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df\": rpc error: code = NotFound desc = could not find container \"eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df\": container with ID starting with eec91b74057fb66deca868c231d1d3242fac2e0d0010a7f23cdc6736ed3758df not found: ID does not exist" Sep 17 17:52:41.438772 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.438735 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s"] Sep 17 17:52:41.440663 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:41.440635 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-76bc4cf9d7-2h22s"] Sep 17 17:52:42.215937 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:42.215905 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" path="/var/lib/kubelet/pods/749b19de-b26d-4b84-97c3-a639fda6e829/volumes" Sep 17 17:52:48.268267 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:48.268178 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 17 17:52:58.269326 ip-10-0-135-40 kubenswrapper[2550]: I0917 17:52:58.269297 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 18:00:35.584643 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.584565 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9"] Sep 17 18:00:35.586975 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.584806 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" containerID="cri-o://0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d" gracePeriod=30 Sep 17 18:00:35.723954 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.723920 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm"] Sep 17 18:00:35.724212 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.724186 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" containerID="cri-o://510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04" gracePeriod=30 Sep 17 18:00:35.724310 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.724249 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kube-rbac-proxy" containerID="cri-o://a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce" gracePeriod=30 Sep 17 18:00:35.839341 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.839312 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48"] Sep 17 18:00:35.839693 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.839680 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" Sep 17 18:00:35.839785 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.839697 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" Sep 17 18:00:35.839785 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.839766 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="749b19de-b26d-4b84-97c3-a639fda6e829" containerName="model-chainer" Sep 17 18:00:35.842850 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.842830 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:35.845485 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.845458 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dcf6-kube-rbac-proxy-sar-config\"" Sep 17 18:00:35.845611 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.845462 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0dcf6-predictor-serving-cert\"" Sep 17 18:00:35.854904 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.854872 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48"] Sep 17 18:00:35.927881 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.927841 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-proxy-tls\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:35.927881 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.927882 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:35.928081 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:35.927905 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27vn\" (UniqueName: \"kubernetes.io/projected/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-api-access-n27vn\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.028635 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.028595 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-proxy-tls\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.028635 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.028632 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.028889 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.028660 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27vn\" (UniqueName: \"kubernetes.io/projected/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-api-access-n27vn\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.029303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.029277 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.031136 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.031116 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-proxy-tls\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.037959 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.037930 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27vn\" (UniqueName: \"kubernetes.io/projected/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-api-access-n27vn\") pod \"success-200-isvc-0dcf6-predictor-5f6f74f858-hls48\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.154880 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.154789 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:36.428080 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.428051 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48"] Sep 17 18:00:36.429940 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:00:36.429914 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e60a16_4e42_40b3_8f56_3af7a0ec10ed.slice/crio-93b4d86de2c66c518cf602a74cce1a461543d11d485e8a39b2c18601c7114e20 WatchSource:0}: Error finding container 93b4d86de2c66c518cf602a74cce1a461543d11d485e8a39b2c18601c7114e20: Status 404 returned error can't find the container with id 93b4d86de2c66c518cf602a74cce1a461543d11d485e8a39b2c18601c7114e20 Sep 17 18:00:36.457780 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.457752 2550 generic.go:356] "Generic (PLEG): container finished" podID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerID="a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce" exitCode=2 Sep 17 18:00:36.457881 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.457828 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" event={"ID":"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4","Type":"ContainerDied","Data":"a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce"} Sep 17 18:00:36.458947 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:36.458928 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" event={"ID":"41e60a16-4e42-40b3-8f56-3af7a0ec10ed","Type":"ContainerStarted","Data":"93b4d86de2c66c518cf602a74cce1a461543d11d485e8a39b2c18601c7114e20"} Sep 17 18:00:37.465091 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:37.465056 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" event={"ID":"41e60a16-4e42-40b3-8f56-3af7a0ec10ed","Type":"ContainerStarted","Data":"4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af"} Sep 17 18:00:37.465478 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:37.465121 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" event={"ID":"41e60a16-4e42-40b3-8f56-3af7a0ec10ed","Type":"ContainerStarted","Data":"b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a"} Sep 17 18:00:37.465478 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:37.465208 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:37.512917 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:37.512863 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podStartSLOduration=2.5128460009999998 podStartE2EDuration="2.512846001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:00:37.512191043 +0000 UTC m=+1183.812758587" watchObservedRunningTime="2026-09-17 18:00:37.512846001 +0000 UTC m=+1183.813413545" Sep 17 18:00:38.036732 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:38.036692 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused" Sep 17 18:00:38.042852 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:38.042816 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 17 18:00:38.313339 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:38.313249 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:00:38.469915 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:38.469875 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:38.471416 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:38.471389 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 17 18:00:38.978504 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:38.978479 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 18:00:39.054348 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.054260 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-rbac-proxy-sar-config\") pod \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " Sep 17 18:00:39.054348 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.054308 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-proxy-tls\") pod \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " Sep 17 18:00:39.054348 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.054346 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp8ln\" (UniqueName: \"kubernetes.io/projected/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-api-access-rp8ln\") pod \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\" (UID: \"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4\") " Sep 17 18:00:39.054667 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.054643 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" (UID: "86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:00:39.056442 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.056416 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-api-access-rp8ln" (OuterVolumeSpecName: "kube-api-access-rp8ln") pod "86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" (UID: "86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4"). InnerVolumeSpecName "kube-api-access-rp8ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:00:39.056505 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.056463 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" (UID: "86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:00:39.155328 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.155285 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8ln\" (UniqueName: \"kubernetes.io/projected/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-api-access-rp8ln\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:00:39.155328 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.155320 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:00:39.155328 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.155333 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:00:39.475578 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.475547 2550 generic.go:356] "Generic (PLEG): container finished" podID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerID="510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04" exitCode=0 Sep 17 18:00:39.476037 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.475593 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" Sep 17 18:00:39.476037 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.475624 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" event={"ID":"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4","Type":"ContainerDied","Data":"510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04"} Sep 17 18:00:39.476037 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.475703 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm" event={"ID":"86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4","Type":"ContainerDied","Data":"ec3cca6fa64414874036bbf684c851261256a47b5cc3bd2566fc9398bbb7a529"} Sep 17 18:00:39.476037 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.475746 2550 scope.go:117] "RemoveContainer" containerID="a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce" Sep 17 18:00:39.476272 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.476251 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 17 18:00:39.484411 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.484395 2550 scope.go:117] "RemoveContainer" containerID="510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04" Sep 17 18:00:39.494208 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.494185 2550 scope.go:117] "RemoveContainer" containerID="a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce" Sep 17 18:00:39.494500 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:00:39.494480 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce\": container with ID starting with a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce not found: ID does not exist" containerID="a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce" Sep 17 18:00:39.494727 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.494508 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce"} err="failed to get container status \"a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce\": rpc error: code = NotFound desc = could not find container \"a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce\": container with ID starting with a36f1c43dbf052758073b592c7c6853625770862b7fffccd005977a054cc7bce not found: ID does not exist" Sep 17 18:00:39.494727 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.494548 2550 scope.go:117] "RemoveContainer" containerID="510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04" Sep 17 18:00:39.494822 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:00:39.494770 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04\": container with ID starting with 510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04 not found: ID does not exist" containerID="510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04" Sep 17 18:00:39.494822 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.494789 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04"} err="failed to get container status \"510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04\": rpc error: code = NotFound desc = could not find container \"510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04\": container with ID starting with 510f9639f32610193a442a1132c91b058e3ed6ee13b251b068b4eacba5c73f04 not found: ID does not exist" Sep 17 18:00:39.497784 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.497761 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm"] Sep 17 18:00:39.501811 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:39.501793 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8592d-predictor-b4d845d84-vrdmm"] Sep 17 18:00:40.215003 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:40.214959 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" path="/var/lib/kubelet/pods/86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4/volumes" Sep 17 18:00:43.313428 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:43.313390 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:00:44.481247 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:44.481218 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:00:44.481709 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:44.481681 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 17 18:00:48.313137 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:48.313099 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:00:48.313632 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:48.313204 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 18:00:53.312782 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:53.312742 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:00:54.482203 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:54.482170 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 17 18:00:58.312885 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:00:58.312835 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:01:03.313201 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:03.313163 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 17 18:01:04.482458 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:04.482419 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 17 18:01:05.730958 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:05.730927 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 18:01:05.879014 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:05.878902 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-proxy-tls\") pod \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " Sep 17 18:01:05.879183 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:05.879022 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-openshift-service-ca-bundle\") pod \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\" (UID: \"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9\") " Sep 17 18:01:05.879390 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:05.879360 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" (UID: "01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:01:05.881034 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:05.881005 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" (UID: "01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:01:05.980501 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:05.980447 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:05.980501 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:05.980495 2550 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9-openshift-service-ca-bundle\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:06.593627 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.593544 2550 generic.go:356] "Generic (PLEG): container finished" podID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerID="0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d" exitCode=0 Sep 17 18:01:06.593627 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.593588 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" Sep 17 18:01:06.593799 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.593623 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" event={"ID":"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9","Type":"ContainerDied","Data":"0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d"} Sep 17 18:01:06.593799 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.593673 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9" event={"ID":"01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9","Type":"ContainerDied","Data":"b0ab305e766842116edb4f4ffe49b28e49b2a8aa60236e5d64a31a891a23f144"} Sep 17 18:01:06.593799 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.593696 2550 scope.go:117] "RemoveContainer" containerID="0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d" Sep 17 18:01:06.604082 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.604048 2550 scope.go:117] "RemoveContainer" containerID="0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d" Sep 17 18:01:06.604566 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:06.604519 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d\": container with ID starting with 0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d not found: ID does not exist" containerID="0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d" Sep 17 18:01:06.604678 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.604577 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d"} err="failed to get container status \"0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d\": rpc error: code = NotFound desc = could not find container \"0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d\": container with ID starting with 0ad2ec5a41625effdda34f50b78bec1f3b40ff9459a4c283ad6d389bdc4cbe8d not found: ID does not exist" Sep 17 18:01:06.610302 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.610270 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9"] Sep 17 18:01:06.615438 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:06.615412 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-8592d-688fc594f9-lcxv9"] Sep 17 18:01:08.215774 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:08.215741 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" path="/var/lib/kubelet/pods/01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9/volumes" Sep 17 18:01:14.482616 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:14.482579 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 17 18:01:24.482749 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:24.482717 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:01:25.916042 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:25.916010 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp"] Sep 17 18:01:25.916448 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:25.916256 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" containerID="cri-o://6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541" gracePeriod=30 Sep 17 18:01:25.916448 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:25.916301 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kube-rbac-proxy" containerID="cri-o://2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296" gracePeriod=30 Sep 17 18:01:26.065957 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.065917 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr"] Sep 17 18:01:26.066318 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066297 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" Sep 17 18:01:26.066318 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066315 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" Sep 17 18:01:26.066466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066326 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kube-rbac-proxy" Sep 17 18:01:26.066466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066332 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kube-rbac-proxy" Sep 17 18:01:26.066466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066349 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" Sep 17 18:01:26.066466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066355 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" Sep 17 18:01:26.066466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066410 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kube-rbac-proxy" Sep 17 18:01:26.066466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066417 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ee9c65-b9ff-4d87-ab12-60bc80b5d9c4" containerName="kserve-container" Sep 17 18:01:26.066466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.066424 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="01585e3c-a3a1-4b0d-b15d-52eedb9ccaa9" containerName="switch-graph-8592d" Sep 17 18:01:26.069517 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.069496 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.071937 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.071915 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8649f-kube-rbac-proxy-sar-config\"" Sep 17 18:01:26.072059 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.072015 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8649f-predictor-serving-cert\"" Sep 17 18:01:26.081758 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.081732 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr"] Sep 17 18:01:26.138017 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.137982 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvv8\" (UniqueName: \"kubernetes.io/projected/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-api-access-swvv8\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.138221 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.138072 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.138221 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.138194 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.238669 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.238582 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.238669 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.238630 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvv8\" (UniqueName: \"kubernetes.io/projected/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-api-access-swvv8\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.238669 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.238657 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.238961 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:26.238761 2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8649f-predictor-serving-cert: secret "success-200-isvc-8649f-predictor-serving-cert" not found Sep 17 18:01:26.238961 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:26.238847 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls podName:0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3 nodeName:}" failed. No retries permitted until 2026-09-17 18:01:26.738825854 +0000 UTC m=+1233.039393394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls") pod "success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" (UID: "0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3") : secret "success-200-isvc-8649f-predictor-serving-cert" not found Sep 17 18:01:26.239311 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.239293 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.250270 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.250236 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvv8\" (UniqueName: \"kubernetes.io/projected/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-api-access-swvv8\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.681346 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.681311 2550 generic.go:356] "Generic (PLEG): container finished" podID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerID="2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296" exitCode=2 Sep 17 18:01:26.681509 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.681376 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" event={"ID":"9df9a881-bfd4-4352-8f03-d0060649fb9d","Type":"ContainerDied","Data":"2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296"} Sep 17 18:01:26.742917 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.742852 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.745611 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.745578 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls\") pod \"success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:26.980746 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:26.980641 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:27.252832 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.252718 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr"] Sep 17 18:01:27.255617 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:01:27.255590 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa7a903_cc6e_45d4_8b34_eb93a0d0cab3.slice/crio-80123e36623dda571e307ac72ea3e3fe58806a6ad18c5aea4a6852f6df81e6e5 WatchSource:0}: Error finding container 80123e36623dda571e307ac72ea3e3fe58806a6ad18c5aea4a6852f6df81e6e5: Status 404 returned error can't find the container with id 80123e36623dda571e307ac72ea3e3fe58806a6ad18c5aea4a6852f6df81e6e5 Sep 17 18:01:27.688778 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.688744 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" event={"ID":"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3","Type":"ContainerStarted","Data":"050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2"} Sep 17 18:01:27.688962 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.688808 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:27.688962 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.688828 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" event={"ID":"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3","Type":"ContainerStarted","Data":"b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b"} Sep 17 18:01:27.688962 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.688847 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:27.688962 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.688864 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" event={"ID":"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3","Type":"ContainerStarted","Data":"80123e36623dda571e307ac72ea3e3fe58806a6ad18c5aea4a6852f6df81e6e5"} Sep 17 18:01:27.690343 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.690315 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 17 18:01:27.710307 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:27.710253 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podStartSLOduration=1.710235572 podStartE2EDuration="1.710235572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:27.707837851 +0000 UTC m=+1234.008405394" watchObservedRunningTime="2026-09-17 18:01:27.710235572 +0000 UTC m=+1234.010803500" Sep 17 18:01:28.262875 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:28.262826 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused" Sep 17 18:01:28.268346 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:28.268319 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 17 18:01:28.693570 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:28.693504 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 17 18:01:29.263037 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.263013 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 18:01:29.366899 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.366806 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df9a881-bfd4-4352-8f03-d0060649fb9d-proxy-tls\") pod \"9df9a881-bfd4-4352-8f03-d0060649fb9d\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " Sep 17 18:01:29.366899 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.366866 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-rbac-proxy-sar-config\") pod \"9df9a881-bfd4-4352-8f03-d0060649fb9d\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " Sep 17 18:01:29.366899 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.366909 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zml7\" (UniqueName: \"kubernetes.io/projected/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-api-access-5zml7\") pod \"9df9a881-bfd4-4352-8f03-d0060649fb9d\" (UID: \"9df9a881-bfd4-4352-8f03-d0060649fb9d\") " Sep 17 18:01:29.367233 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.367203 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9df9a881-bfd4-4352-8f03-d0060649fb9d" (UID: "9df9a881-bfd4-4352-8f03-d0060649fb9d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:01:29.368928 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.368905 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df9a881-bfd4-4352-8f03-d0060649fb9d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9df9a881-bfd4-4352-8f03-d0060649fb9d" (UID: "9df9a881-bfd4-4352-8f03-d0060649fb9d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:01:29.369024 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.369000 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-api-access-5zml7" (OuterVolumeSpecName: "kube-api-access-5zml7") pod "9df9a881-bfd4-4352-8f03-d0060649fb9d" (UID: "9df9a881-bfd4-4352-8f03-d0060649fb9d"). InnerVolumeSpecName "kube-api-access-5zml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:01:29.467798 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.467764 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9df9a881-bfd4-4352-8f03-d0060649fb9d-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:29.467798 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.467790 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:29.467798 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.467799 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zml7\" (UniqueName: \"kubernetes.io/projected/9df9a881-bfd4-4352-8f03-d0060649fb9d-kube-api-access-5zml7\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:29.699496 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.699463 2550 generic.go:356] "Generic (PLEG): container finished" podID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerID="6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541" exitCode=0 Sep 17 18:01:29.699674 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.699517 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" Sep 17 18:01:29.699674 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.699556 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" event={"ID":"9df9a881-bfd4-4352-8f03-d0060649fb9d","Type":"ContainerDied","Data":"6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541"} Sep 17 18:01:29.699674 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.699615 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp" event={"ID":"9df9a881-bfd4-4352-8f03-d0060649fb9d","Type":"ContainerDied","Data":"8daebeaa1be5e0f5ce4a51ddf5215b343bc7c1e9f9fe9f268f3e067d53b33bc3"} Sep 17 18:01:29.699674 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.699641 2550 scope.go:117] "RemoveContainer" containerID="2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296" Sep 17 18:01:29.713917 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.713899 2550 scope.go:117] "RemoveContainer" containerID="6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541" Sep 17 18:01:29.721400 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.721382 2550 scope.go:117] "RemoveContainer" containerID="2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296" Sep 17 18:01:29.721703 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:29.721681 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296\": container with ID starting with 2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296 not found: ID does not exist" containerID="2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296" Sep 17 18:01:29.721780 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.721715 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296"} err="failed to get container status \"2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296\": rpc error: code = NotFound desc = could not find container \"2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296\": container with ID starting with 2102805e5604a0bf4f91f41139aeb863882d583339045d7107730973d32f9296 not found: ID does not exist" Sep 17 18:01:29.721780 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.721739 2550 scope.go:117] "RemoveContainer" containerID="6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541" Sep 17 18:01:29.722045 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:29.722028 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541\": container with ID starting with 6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541 not found: ID does not exist" containerID="6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541" Sep 17 18:01:29.722084 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.722051 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541"} err="failed to get container status \"6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541\": rpc error: code = NotFound desc = could not find container \"6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541\": container with ID starting with 6733b975ba5fab0b07c026460e842f8389632c6feb0bc0c41afdd7c3d49af541 not found: ID does not exist" Sep 17 18:01:29.724857 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.724836 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp"] Sep 17 18:01:29.728120 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:29.728097 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-081d8-predictor-6b56b68db-qlfsp"] Sep 17 18:01:30.215544 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:30.215509 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" path="/var/lib/kubelet/pods/9df9a881-bfd4-4352-8f03-d0060649fb9d/volumes" Sep 17 18:01:33.698192 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:33.698163 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:01:33.698746 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:33.698723 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 17 18:01:43.699624 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:43.699583 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 17 18:01:46.022842 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.022800 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48"] Sep 17 18:01:46.023344 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.023100 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" containerID="cri-o://b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a" gracePeriod=30 Sep 17 18:01:46.023344 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.023159 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kube-rbac-proxy" containerID="cri-o://4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af" gracePeriod=30 Sep 17 18:01:46.147995 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.147962 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl"] Sep 17 18:01:46.148299 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.148288 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kube-rbac-proxy" Sep 17 18:01:46.148344 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.148302 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kube-rbac-proxy" Sep 17 18:01:46.148344 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.148319 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" Sep 17 18:01:46.148344 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.148325 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" Sep 17 18:01:46.148439 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.148379 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kube-rbac-proxy" Sep 17 18:01:46.148439 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.148391 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df9a881-bfd4-4352-8f03-d0060649fb9d" containerName="kserve-container" Sep 17 18:01:46.151231 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.151216 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.155152 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.155132 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9d702-predictor-serving-cert\"" Sep 17 18:01:46.155492 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.155475 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9d702-kube-rbac-proxy-sar-config\"" Sep 17 18:01:46.163413 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.163387 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl"] Sep 17 18:01:46.203790 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.203752 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0235e986-f167-4b11-b51a-65e57f101ae1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.203972 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.203854 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rjc\" (UniqueName: \"kubernetes.io/projected/0235e986-f167-4b11-b51a-65e57f101ae1-kube-api-access-52rjc\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.204153 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.204128 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.304621 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.304510 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.304621 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.304579 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0235e986-f167-4b11-b51a-65e57f101ae1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.304827 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.304643 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rjc\" (UniqueName: \"kubernetes.io/projected/0235e986-f167-4b11-b51a-65e57f101ae1-kube-api-access-52rjc\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.304827 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:46.304663 2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-9d702-predictor-serving-cert: secret "success-200-isvc-9d702-predictor-serving-cert" not found Sep 17 18:01:46.304827 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:46.304744 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls podName:0235e986-f167-4b11-b51a-65e57f101ae1 nodeName:}" failed. No retries permitted until 2026-09-17 18:01:46.804723031 +0000 UTC m=+1253.105290557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls") pod "success-200-isvc-9d702-predictor-84c754796-qj7zl" (UID: "0235e986-f167-4b11-b51a-65e57f101ae1") : secret "success-200-isvc-9d702-predictor-serving-cert" not found Sep 17 18:01:46.305244 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.305225 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0235e986-f167-4b11-b51a-65e57f101ae1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.318726 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.318696 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rjc\" (UniqueName: \"kubernetes.io/projected/0235e986-f167-4b11-b51a-65e57f101ae1-kube-api-access-52rjc\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.777404 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.777365 2550 generic.go:356] "Generic (PLEG): container finished" podID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerID="4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af" exitCode=2 Sep 17 18:01:46.777586 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.777439 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" event={"ID":"41e60a16-4e42-40b3-8f56-3af7a0ec10ed","Type":"ContainerDied","Data":"4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af"} Sep 17 18:01:46.809773 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.809749 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:46.812154 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:46.812128 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls\") pod \"success-200-isvc-9d702-predictor-84c754796-qj7zl\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:47.062014 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:47.061934 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:47.334851 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:47.334777 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl"] Sep 17 18:01:47.338294 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:01:47.338260 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0235e986_f167_4b11_b51a_65e57f101ae1.slice/crio-541ecf8418bb663c85a8f29879081d008b939e03ff311cce855d3f9753101ded WatchSource:0}: Error finding container 541ecf8418bb663c85a8f29879081d008b939e03ff311cce855d3f9753101ded: Status 404 returned error can't find the container with id 541ecf8418bb663c85a8f29879081d008b939e03ff311cce855d3f9753101ded Sep 17 18:01:47.783916 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:47.783876 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" event={"ID":"0235e986-f167-4b11-b51a-65e57f101ae1","Type":"ContainerStarted","Data":"4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762"} Sep 17 18:01:47.784087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:47.783959 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" event={"ID":"0235e986-f167-4b11-b51a-65e57f101ae1","Type":"ContainerStarted","Data":"0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690"} Sep 17 18:01:47.784087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:47.783981 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" event={"ID":"0235e986-f167-4b11-b51a-65e57f101ae1","Type":"ContainerStarted","Data":"541ecf8418bb663c85a8f29879081d008b939e03ff311cce855d3f9753101ded"} Sep 17 18:01:47.784087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:47.784041 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:47.805606 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:47.805556 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podStartSLOduration=1.805523561 podStartE2EDuration="1.805523561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:01:47.803125364 +0000 UTC m=+1254.103692929" watchObservedRunningTime="2026-09-17 18:01:47.805523561 +0000 UTC m=+1254.106091104" Sep 17 18:01:48.788227 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:48.788193 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:48.789826 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:48.789798 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" Sep 17 18:01:49.271726 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.271702 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:01:49.331830 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.331746 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-rbac-proxy-sar-config\") pod \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " Sep 17 18:01:49.331830 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.331779 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n27vn\" (UniqueName: \"kubernetes.io/projected/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-api-access-n27vn\") pod \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " Sep 17 18:01:49.331830 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.331819 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-proxy-tls\") pod \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\" (UID: \"41e60a16-4e42-40b3-8f56-3af7a0ec10ed\") " Sep 17 18:01:49.332159 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.332127 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41e60a16-4e42-40b3-8f56-3af7a0ec10ed" (UID: "41e60a16-4e42-40b3-8f56-3af7a0ec10ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:01:49.333886 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.333859 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-api-access-n27vn" (OuterVolumeSpecName: "kube-api-access-n27vn") pod "41e60a16-4e42-40b3-8f56-3af7a0ec10ed" (UID: "41e60a16-4e42-40b3-8f56-3af7a0ec10ed"). InnerVolumeSpecName "kube-api-access-n27vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:01:49.333886 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.333867 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41e60a16-4e42-40b3-8f56-3af7a0ec10ed" (UID: "41e60a16-4e42-40b3-8f56-3af7a0ec10ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:01:49.433153 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.433105 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:49.433153 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.433147 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27vn\" (UniqueName: \"kubernetes.io/projected/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-kube-api-access-n27vn\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:49.433153 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.433158 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41e60a16-4e42-40b3-8f56-3af7a0ec10ed-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:01:49.794412 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.794378 2550 generic.go:356] "Generic (PLEG): container finished" podID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerID="b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a" exitCode=0 Sep 17 18:01:49.794939 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.794423 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" Sep 17 18:01:49.794939 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.794463 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" event={"ID":"41e60a16-4e42-40b3-8f56-3af7a0ec10ed","Type":"ContainerDied","Data":"b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a"} Sep 17 18:01:49.794939 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.794519 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48" event={"ID":"41e60a16-4e42-40b3-8f56-3af7a0ec10ed","Type":"ContainerDied","Data":"93b4d86de2c66c518cf602a74cce1a461543d11d485e8a39b2c18601c7114e20"} Sep 17 18:01:49.794939 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.794567 2550 scope.go:117] "RemoveContainer" containerID="4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af" Sep 17 18:01:49.795150 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.795114 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" Sep 17 18:01:49.803341 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.803323 2550 scope.go:117] "RemoveContainer" containerID="b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a" Sep 17 18:01:49.810817 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.810800 2550 scope.go:117] "RemoveContainer" containerID="4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af" Sep 17 18:01:49.811130 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:49.811103 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af\": container with ID starting with 4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af not found: ID does not exist" containerID="4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af" Sep 17 18:01:49.811195 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.811138 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af"} err="failed to get container status \"4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af\": rpc error: code = NotFound desc = could not find container \"4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af\": container with ID starting with 4b433c95c25f7f4377ce96e18ac284547ab1f9490aeb0b4c7122a322525e89af not found: ID does not exist" Sep 17 18:01:49.811195 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.811156 2550 scope.go:117] "RemoveContainer" containerID="b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a" Sep 17 18:01:49.811486 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:01:49.811459 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a\": container with ID starting with b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a not found: ID does not exist" containerID="b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a" Sep 17 18:01:49.811585 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.811495 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a"} err="failed to get container status \"b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a\": rpc error: code = NotFound desc = could not find container \"b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a\": container with ID starting with b9f6482b750e0d6c67bcb836dd8a985d31047a3775894c6e5edf3f7d9427b89a not found: ID does not exist" Sep 17 18:01:49.823231 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.823209 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48"] Sep 17 18:01:49.828394 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:49.828373 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0dcf6-predictor-5f6f74f858-hls48"] Sep 17 18:01:50.215391 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:50.215351 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" path="/var/lib/kubelet/pods/41e60a16-4e42-40b3-8f56-3af7a0ec10ed/volumes" Sep 17 18:01:53.699564 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:53.699504 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 17 18:01:54.800046 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:54.800015 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:01:54.800497 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:01:54.800468 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" Sep 17 18:02:03.698815 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:03.698773 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused" Sep 17 18:02:04.801401 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:04.801363 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" Sep 17 18:02:13.700447 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:13.700413 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:02:14.800977 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:14.800937 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" Sep 17 18:02:24.800875 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:24.800828 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused" Sep 17 18:02:34.801384 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:34.801347 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:02:36.181749 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.181719 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr"] Sep 17 18:02:36.182166 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.181943 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" containerID="cri-o://b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b" gracePeriod=30 Sep 17 18:02:36.182166 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.181994 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kube-rbac-proxy" containerID="cri-o://050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2" gracePeriod=30 Sep 17 18:02:36.316383 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.313519 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8"] Sep 17 18:02:36.316383 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.314299 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" Sep 17 18:02:36.316383 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.314319 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" Sep 17 18:02:36.316383 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.314363 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kube-rbac-proxy" Sep 17 18:02:36.316383 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.314374 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kube-rbac-proxy" Sep 17 18:02:36.316383 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.314551 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kube-rbac-proxy" Sep 17 18:02:36.316383 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.314572 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e60a16-4e42-40b3-8f56-3af7a0ec10ed" containerName="kserve-container" Sep 17 18:02:36.318940 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.318918 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.321406 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.321386 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8eede-predictor-serving-cert\"" Sep 17 18:02:36.321543 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.321416 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8eede-kube-rbac-proxy-sar-config\"" Sep 17 18:02:36.326844 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.326822 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8"] Sep 17 18:02:36.422141 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.422097 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.422141 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.422146 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtc7w\" (UniqueName: \"kubernetes.io/projected/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-api-access-rtc7w\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.422376 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.422286 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.522757 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.522672 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtc7w\" (UniqueName: \"kubernetes.io/projected/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-api-access-rtc7w\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.522757 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.522753 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.522992 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.522781 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.522992 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:02:36.522902 2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8eede-predictor-serving-cert: secret "success-200-isvc-8eede-predictor-serving-cert" not found Sep 17 18:02:36.523096 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:02:36.522998 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls podName:9bf80e37-d9f6-4ce8-8666-14bdd0b681ce nodeName:}" failed. No retries permitted until 2026-09-17 18:02:37.022969847 +0000 UTC m=+1303.323537392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls") pod "success-200-isvc-8eede-predictor-757747b4bb-fm7b8" (UID: "9bf80e37-d9f6-4ce8-8666-14bdd0b681ce") : secret "success-200-isvc-8eede-predictor-serving-cert" not found Sep 17 18:02:36.523416 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.523397 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:36.532454 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:36.532434 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtc7w\" (UniqueName: \"kubernetes.io/projected/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-api-access-rtc7w\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:37.003177 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:37.003140 2550 generic.go:356] "Generic (PLEG): container finished" podID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerID="050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2" exitCode=2 Sep 17 18:02:37.003356 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:37.003215 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" event={"ID":"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3","Type":"ContainerDied","Data":"050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2"} Sep 17 18:02:37.027236 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:37.027200 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:37.029606 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:37.029579 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls\") pod \"success-200-isvc-8eede-predictor-757747b4bb-fm7b8\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:37.230105 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:37.230070 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:37.514490 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:37.514452 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8"] Sep 17 18:02:37.517326 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:02:37.517296 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf80e37_d9f6_4ce8_8666_14bdd0b681ce.slice/crio-ae54bfa5cd569ddc7d0382ae15f110a791ae9df54e91cd665d0761bc338a255c WatchSource:0}: Error finding container ae54bfa5cd569ddc7d0382ae15f110a791ae9df54e91cd665d0761bc338a255c: Status 404 returned error can't find the container with id ae54bfa5cd569ddc7d0382ae15f110a791ae9df54e91cd665d0761bc338a255c Sep 17 18:02:38.010079 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:38.010046 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" event={"ID":"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce","Type":"ContainerStarted","Data":"f555d981b5b50872e1a98b964c04b66184b588f2696fac3a404fd67a15c343fd"} Sep 17 18:02:38.010257 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:38.010100 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" event={"ID":"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce","Type":"ContainerStarted","Data":"c96718e71eccad5f354c426b631bae3007e9b7c9108f6bf948d3ec8e133fe53e"} Sep 17 18:02:38.010257 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:38.010121 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" event={"ID":"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce","Type":"ContainerStarted","Data":"ae54bfa5cd569ddc7d0382ae15f110a791ae9df54e91cd665d0761bc338a255c"} Sep 17 18:02:38.010257 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:38.010203 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:38.030728 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:38.030680 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podStartSLOduration=2.030661765 podStartE2EDuration="2.030661765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:38.030017837 +0000 UTC m=+1304.330585381" watchObservedRunningTime="2026-09-17 18:02:38.030661765 +0000 UTC m=+1304.331229309" Sep 17 18:02:38.694578 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:38.694510 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused" Sep 17 18:02:39.014045 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.013954 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:39.015290 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.015263 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 17 18:02:39.432902 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.432876 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:02:39.448486 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.448457 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swvv8\" (UniqueName: \"kubernetes.io/projected/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-api-access-swvv8\") pod \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " Sep 17 18:02:39.448642 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.448564 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-rbac-proxy-sar-config\") pod \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " Sep 17 18:02:39.448642 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.448598 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls\") pod \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\" (UID: \"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3\") " Sep 17 18:02:39.448901 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.448876 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" (UID: "0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:02:39.450699 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.450668 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" (UID: "0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:02:39.450699 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.450674 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-api-access-swvv8" (OuterVolumeSpecName: "kube-api-access-swvv8") pod "0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" (UID: "0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3"). InnerVolumeSpecName "kube-api-access-swvv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:02:39.550188 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.550098 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:02:39.550188 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.550130 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:02:39.550188 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:39.550141 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvv8\" (UniqueName: \"kubernetes.io/projected/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3-kube-api-access-swvv8\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:02:40.020540 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.020504 2550 generic.go:356] "Generic (PLEG): container finished" podID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerID="b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b" exitCode=0 Sep 17 18:02:40.021106 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.020574 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" Sep 17 18:02:40.021106 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.020589 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" event={"ID":"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3","Type":"ContainerDied","Data":"b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b"} Sep 17 18:02:40.021106 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.020642 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr" event={"ID":"0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3","Type":"ContainerDied","Data":"80123e36623dda571e307ac72ea3e3fe58806a6ad18c5aea4a6852f6df81e6e5"} Sep 17 18:02:40.021106 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.020669 2550 scope.go:117] "RemoveContainer" containerID="050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2" Sep 17 18:02:40.021106 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.021088 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 17 18:02:40.029665 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.029649 2550 scope.go:117] "RemoveContainer" containerID="b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b" Sep 17 18:02:40.036765 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.036749 2550 scope.go:117] "RemoveContainer" containerID="050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2" Sep 17 18:02:40.037010 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:02:40.036992 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2\": container with ID starting with 050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2 not found: ID does not exist" containerID="050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2" Sep 17 18:02:40.037067 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.037017 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2"} err="failed to get container status \"050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2\": rpc error: code = NotFound desc = could not find container \"050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2\": container with ID starting with 050b58a4ffe88842bcc584cc4c5eee9dbde3ada3c29aefe31d44219a691e95d2 not found: ID does not exist" Sep 17 18:02:40.037067 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.037034 2550 scope.go:117] "RemoveContainer" containerID="b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b" Sep 17 18:02:40.037271 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:02:40.037251 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b\": container with ID starting with b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b not found: ID does not exist" containerID="b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b" Sep 17 18:02:40.037313 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.037277 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b"} err="failed to get container status \"b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b\": rpc error: code = NotFound desc = could not find container \"b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b\": container with ID starting with b630c3e18e2b7315aaa04c4490e3b78c97d963698f661c328e75b167840f2e3b not found: ID does not exist" Sep 17 18:02:40.047791 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.047759 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr"] Sep 17 18:02:40.056219 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.056196 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8649f-predictor-86d8f8bc54-9q6dr"] Sep 17 18:02:40.215414 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:40.215380 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" path="/var/lib/kubelet/pods/0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3/volumes" Sep 17 18:02:45.025823 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:45.025791 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:02:45.026402 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:45.026377 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 17 18:02:55.026498 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:02:55.026461 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 17 18:03:05.026651 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:03:05.026611 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 17 18:03:15.026804 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:03:15.026762 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 17 18:03:25.028369 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:03:25.028289 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:11:00.900773 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:00.900739 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl"] Sep 17 18:11:00.901252 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:00.900985 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" containerID="cri-o://0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690" gracePeriod=30 Sep 17 18:11:00.901252 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:00.901029 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kube-rbac-proxy" containerID="cri-o://4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762" gracePeriod=30 Sep 17 18:11:01.008684 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.008646 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2"] Sep 17 18:11:01.009035 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.009022 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" Sep 17 18:11:01.009087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.009036 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" Sep 17 18:11:01.009087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.009053 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kube-rbac-proxy" Sep 17 18:11:01.009087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.009059 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kube-rbac-proxy" Sep 17 18:11:01.009191 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.009110 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kube-rbac-proxy" Sep 17 18:11:01.009191 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.009119 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0aa7a903-cc6e-45d4-8b34-eb93a0d0cab3" containerName="kserve-container" Sep 17 18:11:01.012366 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.012346 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.014866 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.014839 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-de19c-predictor-serving-cert\"" Sep 17 18:11:01.014978 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.014908 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-de19c-kube-rbac-proxy-sar-config\"" Sep 17 18:11:01.023630 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.023608 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2"] Sep 17 18:11:01.043956 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.043923 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpgv\" (UniqueName: \"kubernetes.io/projected/2c615fa2-4831-444f-a916-0dfd927210ed-kube-api-access-4zpgv\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.044120 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.044050 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c615fa2-4831-444f-a916-0dfd927210ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.044120 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.044082 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c615fa2-4831-444f-a916-0dfd927210ed-proxy-tls\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.144524 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.144483 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c615fa2-4831-444f-a916-0dfd927210ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.144713 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.144580 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c615fa2-4831-444f-a916-0dfd927210ed-proxy-tls\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.144713 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.144642 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpgv\" (UniqueName: \"kubernetes.io/projected/2c615fa2-4831-444f-a916-0dfd927210ed-kube-api-access-4zpgv\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.145256 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.145235 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c615fa2-4831-444f-a916-0dfd927210ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.147476 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.147448 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c615fa2-4831-444f-a916-0dfd927210ed-proxy-tls\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.154476 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.154422 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpgv\" (UniqueName: \"kubernetes.io/projected/2c615fa2-4831-444f-a916-0dfd927210ed-kube-api-access-4zpgv\") pod \"success-200-isvc-de19c-predictor-548b8d5b44-6g7k2\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.198679 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.198638 2550 generic.go:356] "Generic (PLEG): container finished" podID="0235e986-f167-4b11-b51a-65e57f101ae1" containerID="4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762" exitCode=2 Sep 17 18:11:01.198819 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.198697 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" event={"ID":"0235e986-f167-4b11-b51a-65e57f101ae1","Type":"ContainerDied","Data":"4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762"} Sep 17 18:11:01.323777 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.323736 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:01.599654 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:01.599626 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2"] Sep 17 18:11:01.602130 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:11:01.602101 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c615fa2_4831_444f_a916_0dfd927210ed.slice/crio-f6cb8c1694046096ded385966c5feca473173b13d58b558b03d07cc5ffd9099d WatchSource:0}: Error finding container f6cb8c1694046096ded385966c5feca473173b13d58b558b03d07cc5ffd9099d: Status 404 returned error can't find the container with id f6cb8c1694046096ded385966c5feca473173b13d58b558b03d07cc5ffd9099d Sep 17 18:11:02.205623 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:02.205581 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" event={"ID":"2c615fa2-4831-444f-a916-0dfd927210ed","Type":"ContainerStarted","Data":"e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f"} Sep 17 18:11:02.206039 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:02.205657 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" event={"ID":"2c615fa2-4831-444f-a916-0dfd927210ed","Type":"ContainerStarted","Data":"e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d"} Sep 17 18:11:02.206039 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:02.205685 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" event={"ID":"2c615fa2-4831-444f-a916-0dfd927210ed","Type":"ContainerStarted","Data":"f6cb8c1694046096ded385966c5feca473173b13d58b558b03d07cc5ffd9099d"} Sep 17 18:11:02.206039 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:02.205721 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:02.228432 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:02.228380 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podStartSLOduration=2.228361946 podStartE2EDuration="2.228361946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:02.226469209 +0000 UTC m=+1808.527036754" watchObservedRunningTime="2026-09-17 18:11:02.228361946 +0000 UTC m=+1808.528929491" Sep 17 18:11:03.211359 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:03.211321 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:03.212898 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:03.212873 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 17 18:11:04.169225 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.169200 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:11:04.217488 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.217417 2550 generic.go:356] "Generic (PLEG): container finished" podID="0235e986-f167-4b11-b51a-65e57f101ae1" containerID="0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690" exitCode=0 Sep 17 18:11:04.217488 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.217461 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" Sep 17 18:11:04.217972 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.217493 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" event={"ID":"0235e986-f167-4b11-b51a-65e57f101ae1","Type":"ContainerDied","Data":"0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690"} Sep 17 18:11:04.217972 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.217559 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl" event={"ID":"0235e986-f167-4b11-b51a-65e57f101ae1","Type":"ContainerDied","Data":"541ecf8418bb663c85a8f29879081d008b939e03ff311cce855d3f9753101ded"} Sep 17 18:11:04.217972 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.217585 2550 scope.go:117] "RemoveContainer" containerID="4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762" Sep 17 18:11:04.217972 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.217852 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 17 18:11:04.225465 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.225448 2550 scope.go:117] "RemoveContainer" containerID="0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690" Sep 17 18:11:04.232566 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.232548 2550 scope.go:117] "RemoveContainer" containerID="4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762" Sep 17 18:11:04.232830 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:11:04.232812 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762\": container with ID starting with 4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762 not found: ID does not exist" containerID="4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762" Sep 17 18:11:04.232880 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.232840 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762"} err="failed to get container status \"4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762\": rpc error: code = NotFound desc = could not find container \"4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762\": container with ID starting with 4135c3ad7a2614f6d883d44c3c6c1859ad733f33f3696ea87c60c33971a27762 not found: ID does not exist" Sep 17 18:11:04.232880 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.232858 2550 scope.go:117] "RemoveContainer" containerID="0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690" Sep 17 18:11:04.233094 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:11:04.233078 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690\": container with ID starting with 0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690 not found: ID does not exist" containerID="0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690" Sep 17 18:11:04.233131 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.233098 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690"} err="failed to get container status \"0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690\": rpc error: code = NotFound desc = could not find container \"0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690\": container with ID starting with 0f04b3b8f9c00f4ba6d2c8d73ba2204b0f234bdca54a2132bb4e83f69b1fa690 not found: ID does not exist" Sep 17 18:11:04.266768 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.266741 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0235e986-f167-4b11-b51a-65e57f101ae1-kube-rbac-proxy-sar-config\") pod \"0235e986-f167-4b11-b51a-65e57f101ae1\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " Sep 17 18:11:04.266893 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.266779 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls\") pod \"0235e986-f167-4b11-b51a-65e57f101ae1\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " Sep 17 18:11:04.266893 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.266858 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52rjc\" (UniqueName: \"kubernetes.io/projected/0235e986-f167-4b11-b51a-65e57f101ae1-kube-api-access-52rjc\") pod \"0235e986-f167-4b11-b51a-65e57f101ae1\" (UID: \"0235e986-f167-4b11-b51a-65e57f101ae1\") " Sep 17 18:11:04.267152 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.267107 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0235e986-f167-4b11-b51a-65e57f101ae1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0235e986-f167-4b11-b51a-65e57f101ae1" (UID: "0235e986-f167-4b11-b51a-65e57f101ae1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:11:04.267445 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.267369 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0235e986-f167-4b11-b51a-65e57f101ae1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:11:04.268977 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.268955 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0235e986-f167-4b11-b51a-65e57f101ae1" (UID: "0235e986-f167-4b11-b51a-65e57f101ae1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:11:04.269062 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.268994 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0235e986-f167-4b11-b51a-65e57f101ae1-kube-api-access-52rjc" (OuterVolumeSpecName: "kube-api-access-52rjc") pod "0235e986-f167-4b11-b51a-65e57f101ae1" (UID: "0235e986-f167-4b11-b51a-65e57f101ae1"). InnerVolumeSpecName "kube-api-access-52rjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:11:04.367768 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.367729 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52rjc\" (UniqueName: \"kubernetes.io/projected/0235e986-f167-4b11-b51a-65e57f101ae1-kube-api-access-52rjc\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:11:04.367768 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.367759 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0235e986-f167-4b11-b51a-65e57f101ae1-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:11:04.539224 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.539181 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl"] Sep 17 18:11:04.542967 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:04.542934 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9d702-predictor-84c754796-qj7zl"] Sep 17 18:11:06.215059 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:06.215023 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" path="/var/lib/kubelet/pods/0235e986-f167-4b11-b51a-65e57f101ae1/volumes" Sep 17 18:11:09.221961 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:09.221933 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:09.222473 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:09.222405 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 17 18:11:19.222889 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:19.222826 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 17 18:11:29.223286 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:29.223243 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 17 18:11:39.223248 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:39.223202 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 17 18:11:49.223329 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:49.223294 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:11:51.053466 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.053430 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8"] Sep 17 18:11:51.053894 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.053737 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" containerID="cri-o://c96718e71eccad5f354c426b631bae3007e9b7c9108f6bf948d3ec8e133fe53e" gracePeriod=30 Sep 17 18:11:51.053894 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.053759 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kube-rbac-proxy" containerID="cri-o://f555d981b5b50872e1a98b964c04b66184b588f2696fac3a404fd67a15c343fd" gracePeriod=30 Sep 17 18:11:51.192126 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.192090 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg"] Sep 17 18:11:51.192433 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.192421 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kube-rbac-proxy" Sep 17 18:11:51.192477 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.192435 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kube-rbac-proxy" Sep 17 18:11:51.192477 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.192458 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" Sep 17 18:11:51.192477 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.192464 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" Sep 17 18:11:51.192599 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.192512 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kserve-container" Sep 17 18:11:51.192599 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.192521 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="0235e986-f167-4b11-b51a-65e57f101ae1" containerName="kube-rbac-proxy" Sep 17 18:11:51.195428 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.195407 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.200891 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.200864 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-85bfa-predictor-serving-cert\"" Sep 17 18:11:51.201165 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.201147 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-85bfa-kube-rbac-proxy-sar-config\"" Sep 17 18:11:51.210100 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.210068 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg"] Sep 17 18:11:51.365867 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.365785 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vl5v\" (UniqueName: \"kubernetes.io/projected/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-api-access-4vl5v\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.365867 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.365838 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-proxy-tls\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.366057 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.365874 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.426624 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.426585 2550 generic.go:356] "Generic (PLEG): container finished" podID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerID="f555d981b5b50872e1a98b964c04b66184b588f2696fac3a404fd67a15c343fd" exitCode=2 Sep 17 18:11:51.426804 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.426652 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" event={"ID":"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce","Type":"ContainerDied","Data":"f555d981b5b50872e1a98b964c04b66184b588f2696fac3a404fd67a15c343fd"} Sep 17 18:11:51.467172 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.467136 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vl5v\" (UniqueName: \"kubernetes.io/projected/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-api-access-4vl5v\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.467336 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.467181 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-proxy-tls\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.467336 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.467239 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.467927 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.467906 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.469778 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.469755 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-proxy-tls\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.475623 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.475601 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vl5v\" (UniqueName: \"kubernetes.io/projected/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-api-access-4vl5v\") pod \"success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.512123 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.512091 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:51.786353 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:51.786295 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg"] Sep 17 18:11:51.788552 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:11:51.788511 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f44c675_ab1a_4cc8_80d0_6bbebdcee50a.slice/crio-5d31f2afeb0461ee17c2aa0084e604602089c0aea4adad320cd8d2154255158b WatchSource:0}: Error finding container 5d31f2afeb0461ee17c2aa0084e604602089c0aea4adad320cd8d2154255158b: Status 404 returned error can't find the container with id 5d31f2afeb0461ee17c2aa0084e604602089c0aea4adad320cd8d2154255158b Sep 17 18:11:52.432636 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:52.432602 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" event={"ID":"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a","Type":"ContainerStarted","Data":"68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb"} Sep 17 18:11:52.433039 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:52.432662 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" event={"ID":"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a","Type":"ContainerStarted","Data":"44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d"} Sep 17 18:11:52.433039 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:52.432692 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:52.433039 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:52.432711 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" event={"ID":"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a","Type":"ContainerStarted","Data":"5d31f2afeb0461ee17c2aa0084e604602089c0aea4adad320cd8d2154255158b"} Sep 17 18:11:52.452444 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:52.452380 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podStartSLOduration=1.452358292 podStartE2EDuration="1.452358292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:52.45027419 +0000 UTC m=+1858.750841734" watchObservedRunningTime="2026-09-17 18:11:52.452358292 +0000 UTC m=+1858.752925838" Sep 17 18:11:53.437819 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:53.437778 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:53.439379 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:53.439351 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 17 18:11:54.212213 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.212182 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:11:54.367257 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.367145 2550 scope.go:117] "RemoveContainer" containerID="f555d981b5b50872e1a98b964c04b66184b588f2696fac3a404fd67a15c343fd" Sep 17 18:11:54.375510 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.375486 2550 scope.go:117] "RemoveContainer" containerID="c96718e71eccad5f354c426b631bae3007e9b7c9108f6bf948d3ec8e133fe53e" Sep 17 18:11:54.392237 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.392210 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtc7w\" (UniqueName: \"kubernetes.io/projected/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-api-access-rtc7w\") pod \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " Sep 17 18:11:54.392347 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.392243 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-rbac-proxy-sar-config\") pod \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " Sep 17 18:11:54.392347 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.392272 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls\") pod \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\" (UID: \"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce\") " Sep 17 18:11:54.392642 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.392613 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" (UID: "9bf80e37-d9f6-4ce8-8666-14bdd0b681ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:11:54.394392 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.394368 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" (UID: "9bf80e37-d9f6-4ce8-8666-14bdd0b681ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:11:54.394480 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.394407 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-api-access-rtc7w" (OuterVolumeSpecName: "kube-api-access-rtc7w") pod "9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" (UID: "9bf80e37-d9f6-4ce8-8666-14bdd0b681ce"). InnerVolumeSpecName "kube-api-access-rtc7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:11:54.442564 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.442519 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" Sep 17 18:11:54.443035 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.442512 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" event={"ID":"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce","Type":"ContainerDied","Data":"c96718e71eccad5f354c426b631bae3007e9b7c9108f6bf948d3ec8e133fe53e"} Sep 17 18:11:54.443035 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.442689 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8" event={"ID":"9bf80e37-d9f6-4ce8-8666-14bdd0b681ce","Type":"ContainerDied","Data":"ae54bfa5cd569ddc7d0382ae15f110a791ae9df54e91cd665d0761bc338a255c"} Sep 17 18:11:54.443128 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.443037 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 17 18:11:54.477266 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.477233 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8"] Sep 17 18:11:54.483357 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.483328 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8eede-predictor-757747b4bb-fm7b8"] Sep 17 18:11:54.493018 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.492998 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtc7w\" (UniqueName: \"kubernetes.io/projected/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-api-access-rtc7w\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:11:54.493077 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.493024 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:11:54.493077 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:54.493035 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:11:56.215784 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:56.215747 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" path="/var/lib/kubelet/pods/9bf80e37-d9f6-4ce8-8666-14bdd0b681ce/volumes" Sep 17 18:11:59.447369 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:59.447339 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:11:59.447805 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:11:59.447774 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 17 18:12:09.448702 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:09.448658 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 17 18:12:11.238063 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.238020 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2"] Sep 17 18:12:11.238576 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.238300 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" containerID="cri-o://e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d" gracePeriod=30 Sep 17 18:12:11.238576 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.238368 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kube-rbac-proxy" containerID="cri-o://e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f" gracePeriod=30 Sep 17 18:12:11.353653 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.353622 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss"] Sep 17 18:12:11.353963 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.353951 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kube-rbac-proxy" Sep 17 18:12:11.354005 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.353965 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kube-rbac-proxy" Sep 17 18:12:11.354005 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.353976 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" Sep 17 18:12:11.354005 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.353982 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" Sep 17 18:12:11.354135 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.354040 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kserve-container" Sep 17 18:12:11.354135 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.354051 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf80e37-d9f6-4ce8-8666-14bdd0b681ce" containerName="kube-rbac-proxy" Sep 17 18:12:11.357384 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.357365 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.359823 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.359798 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e7d45-kube-rbac-proxy-sar-config\"" Sep 17 18:12:11.359926 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.359795 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e7d45-predictor-serving-cert\"" Sep 17 18:12:11.366484 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.366458 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss"] Sep 17 18:12:11.430408 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.430370 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.430585 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.430414 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.430585 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.430520 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nhr\" (UniqueName: \"kubernetes.io/projected/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-api-access-w7nhr\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.516791 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.516716 2550 generic.go:356] "Generic (PLEG): container finished" podID="2c615fa2-4831-444f-a916-0dfd927210ed" containerID="e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f" exitCode=2 Sep 17 18:12:11.516928 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.516789 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" event={"ID":"2c615fa2-4831-444f-a916-0dfd927210ed","Type":"ContainerDied","Data":"e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f"} Sep 17 18:12:11.531355 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.531330 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.531489 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.531378 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.531489 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.531440 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nhr\" (UniqueName: \"kubernetes.io/projected/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-api-access-w7nhr\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.531623 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:12:11.531499 2550 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-serving-cert: secret "success-200-isvc-e7d45-predictor-serving-cert" not found Sep 17 18:12:11.531623 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:12:11.531602 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls podName:9f7c7a86-c19a-4817-972e-05b890b9e0ed nodeName:}" failed. No retries permitted until 2026-09-17 18:12:12.031578406 +0000 UTC m=+1878.332145935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls") pod "success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" (UID: "9f7c7a86-c19a-4817-972e-05b890b9e0ed") : secret "success-200-isvc-e7d45-predictor-serving-cert" not found Sep 17 18:12:11.532116 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.532099 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:11.540347 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:11.540329 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nhr\" (UniqueName: \"kubernetes.io/projected/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-api-access-w7nhr\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:12.035675 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:12.035632 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:12.038432 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:12.038395 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls\") pod \"success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:12.268628 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:12.268586 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:12.541315 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:12.541285 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss"] Sep 17 18:12:12.544515 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:12:12.544484 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7c7a86_c19a_4817_972e_05b890b9e0ed.slice/crio-f17970e6466e96bbb3279253a80db23e6229b1309e61a1b5783aa81302a175d1 WatchSource:0}: Error finding container f17970e6466e96bbb3279253a80db23e6229b1309e61a1b5783aa81302a175d1: Status 404 returned error can't find the container with id f17970e6466e96bbb3279253a80db23e6229b1309e61a1b5783aa81302a175d1 Sep 17 18:12:13.527239 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:13.527192 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" event={"ID":"9f7c7a86-c19a-4817-972e-05b890b9e0ed","Type":"ContainerStarted","Data":"9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b"} Sep 17 18:12:13.527669 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:13.527265 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:13.527669 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:13.527296 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:13.527669 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:13.527315 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" event={"ID":"9f7c7a86-c19a-4817-972e-05b890b9e0ed","Type":"ContainerStarted","Data":"61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e"} Sep 17 18:12:13.527669 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:13.527335 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" event={"ID":"9f7c7a86-c19a-4817-972e-05b890b9e0ed","Type":"ContainerStarted","Data":"f17970e6466e96bbb3279253a80db23e6229b1309e61a1b5783aa81302a175d1"} Sep 17 18:12:13.528843 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:13.528809 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 17 18:12:13.546332 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:13.546267 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podStartSLOduration=2.546242369 podStartE2EDuration="2.546242369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:12:13.543841611 +0000 UTC m=+1879.844409156" watchObservedRunningTime="2026-09-17 18:12:13.546242369 +0000 UTC m=+1879.846809916" Sep 17 18:12:14.218347 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:14.218311 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused" Sep 17 18:12:14.531178 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:14.531092 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 17 18:12:15.089658 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.089632 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:12:15.165582 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.165466 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c615fa2-4831-444f-a916-0dfd927210ed-proxy-tls\") pod \"2c615fa2-4831-444f-a916-0dfd927210ed\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " Sep 17 18:12:15.165741 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.165671 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zpgv\" (UniqueName: \"kubernetes.io/projected/2c615fa2-4831-444f-a916-0dfd927210ed-kube-api-access-4zpgv\") pod \"2c615fa2-4831-444f-a916-0dfd927210ed\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " Sep 17 18:12:15.165741 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.165692 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c615fa2-4831-444f-a916-0dfd927210ed-kube-rbac-proxy-sar-config\") pod \"2c615fa2-4831-444f-a916-0dfd927210ed\" (UID: \"2c615fa2-4831-444f-a916-0dfd927210ed\") " Sep 17 18:12:15.166100 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.166072 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c615fa2-4831-444f-a916-0dfd927210ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c615fa2-4831-444f-a916-0dfd927210ed" (UID: "2c615fa2-4831-444f-a916-0dfd927210ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:12:15.167554 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.167509 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c615fa2-4831-444f-a916-0dfd927210ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c615fa2-4831-444f-a916-0dfd927210ed" (UID: "2c615fa2-4831-444f-a916-0dfd927210ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:12:15.167657 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.167636 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c615fa2-4831-444f-a916-0dfd927210ed-kube-api-access-4zpgv" (OuterVolumeSpecName: "kube-api-access-4zpgv") pod "2c615fa2-4831-444f-a916-0dfd927210ed" (UID: "2c615fa2-4831-444f-a916-0dfd927210ed"). InnerVolumeSpecName "kube-api-access-4zpgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:12:15.266774 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.266732 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c615fa2-4831-444f-a916-0dfd927210ed-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:12:15.266774 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.266764 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpgv\" (UniqueName: \"kubernetes.io/projected/2c615fa2-4831-444f-a916-0dfd927210ed-kube-api-access-4zpgv\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:12:15.266774 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.266773 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c615fa2-4831-444f-a916-0dfd927210ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:12:15.537425 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.537332 2550 generic.go:356] "Generic (PLEG): container finished" podID="2c615fa2-4831-444f-a916-0dfd927210ed" containerID="e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d" exitCode=0 Sep 17 18:12:15.537425 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.537384 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" Sep 17 18:12:15.537914 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.537420 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" event={"ID":"2c615fa2-4831-444f-a916-0dfd927210ed","Type":"ContainerDied","Data":"e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d"} Sep 17 18:12:15.537914 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.537481 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2" event={"ID":"2c615fa2-4831-444f-a916-0dfd927210ed","Type":"ContainerDied","Data":"f6cb8c1694046096ded385966c5feca473173b13d58b558b03d07cc5ffd9099d"} Sep 17 18:12:15.537914 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.537507 2550 scope.go:117] "RemoveContainer" containerID="e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f" Sep 17 18:12:15.546790 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.546774 2550 scope.go:117] "RemoveContainer" containerID="e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d" Sep 17 18:12:15.554440 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.554419 2550 scope.go:117] "RemoveContainer" containerID="e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f" Sep 17 18:12:15.554744 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:12:15.554725 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f\": container with ID starting with e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f not found: ID does not exist" containerID="e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f" Sep 17 18:12:15.554807 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.554754 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f"} err="failed to get container status \"e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f\": rpc error: code = NotFound desc = could not find container \"e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f\": container with ID starting with e26168833f50f0813b5cc6212e219d1001a248f2c898123b5b724b2d7eb3150f not found: ID does not exist" Sep 17 18:12:15.554807 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.554774 2550 scope.go:117] "RemoveContainer" containerID="e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d" Sep 17 18:12:15.555009 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:12:15.554995 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d\": container with ID starting with e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d not found: ID does not exist" containerID="e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d" Sep 17 18:12:15.555045 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.555012 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d"} err="failed to get container status \"e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d\": rpc error: code = NotFound desc = could not find container \"e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d\": container with ID starting with e04d435141d9cc65c4bc84598cf313d09397f4d942e17c3ae287cbd9e4627c5d not found: ID does not exist" Sep 17 18:12:15.560162 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.560141 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2"] Sep 17 18:12:15.565739 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:15.565719 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19c-predictor-548b8d5b44-6g7k2"] Sep 17 18:12:16.215806 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:16.215728 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" path="/var/lib/kubelet/pods/2c615fa2-4831-444f-a916-0dfd927210ed/volumes" Sep 17 18:12:19.447893 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:19.447852 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 17 18:12:19.535883 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:19.535855 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:12:19.536288 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:19.536264 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 17 18:12:29.448347 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:29.448303 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 17 18:12:29.537183 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:29.537142 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 17 18:12:39.449676 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:39.449645 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:12:39.536297 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:39.536257 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 17 18:12:49.536381 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:49.536343 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 17 18:12:59.536722 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:12:59.536691 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:21:26.090426 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:26.090272 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss"] Sep 17 18:21:26.092871 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:26.090603 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" containerID="cri-o://61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e" gracePeriod=30 Sep 17 18:21:26.092871 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:26.090774 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kube-rbac-proxy" containerID="cri-o://9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b" gracePeriod=30 Sep 17 18:21:26.936900 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:26.936868 2550 generic.go:356] "Generic (PLEG): container finished" podID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerID="9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b" exitCode=2 Sep 17 18:21:26.937062 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:26.936932 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" event={"ID":"9f7c7a86-c19a-4817-972e-05b890b9e0ed","Type":"ContainerDied","Data":"9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b"} Sep 17 18:21:29.241799 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.241768 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:21:29.311572 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.311465 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7nhr\" (UniqueName: \"kubernetes.io/projected/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-api-access-w7nhr\") pod \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " Sep 17 18:21:29.311572 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.311567 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls\") pod \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " Sep 17 18:21:29.311779 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.311594 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-rbac-proxy-sar-config\") pod \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\" (UID: \"9f7c7a86-c19a-4817-972e-05b890b9e0ed\") " Sep 17 18:21:29.311978 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.311954 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f7c7a86-c19a-4817-972e-05b890b9e0ed" (UID: "9f7c7a86-c19a-4817-972e-05b890b9e0ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:21:29.313765 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.313742 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f7c7a86-c19a-4817-972e-05b890b9e0ed" (UID: "9f7c7a86-c19a-4817-972e-05b890b9e0ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:21:29.313850 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.313819 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-api-access-w7nhr" (OuterVolumeSpecName: "kube-api-access-w7nhr") pod "9f7c7a86-c19a-4817-972e-05b890b9e0ed" (UID: "9f7c7a86-c19a-4817-972e-05b890b9e0ed"). InnerVolumeSpecName "kube-api-access-w7nhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:21:29.412095 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.412052 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f7c7a86-c19a-4817-972e-05b890b9e0ed-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:21:29.412095 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.412088 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:21:29.412095 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.412101 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nhr\" (UniqueName: \"kubernetes.io/projected/9f7c7a86-c19a-4817-972e-05b890b9e0ed-kube-api-access-w7nhr\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:21:29.951487 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.951455 2550 generic.go:356] "Generic (PLEG): container finished" podID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerID="61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e" exitCode=0 Sep 17 18:21:29.951722 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.951509 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" Sep 17 18:21:29.951722 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.951562 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" event={"ID":"9f7c7a86-c19a-4817-972e-05b890b9e0ed","Type":"ContainerDied","Data":"61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e"} Sep 17 18:21:29.951722 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.951633 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss" event={"ID":"9f7c7a86-c19a-4817-972e-05b890b9e0ed","Type":"ContainerDied","Data":"f17970e6466e96bbb3279253a80db23e6229b1309e61a1b5783aa81302a175d1"} Sep 17 18:21:29.951722 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.951671 2550 scope.go:117] "RemoveContainer" containerID="9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b" Sep 17 18:21:29.960167 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.960141 2550 scope.go:117] "RemoveContainer" containerID="61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e" Sep 17 18:21:29.967673 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.967654 2550 scope.go:117] "RemoveContainer" containerID="9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b" Sep 17 18:21:29.967970 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:21:29.967953 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b\": container with ID starting with 9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b not found: ID does not exist" containerID="9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b" Sep 17 18:21:29.968017 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.967980 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b"} err="failed to get container status \"9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b\": rpc error: code = NotFound desc = could not find container \"9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b\": container with ID starting with 9a1b56a345919f9b9987f558675545febef83887bb6bfdc2b2c08223317e123b not found: ID does not exist" Sep 17 18:21:29.968017 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.967996 2550 scope.go:117] "RemoveContainer" containerID="61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e" Sep 17 18:21:29.968231 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:21:29.968209 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e\": container with ID starting with 61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e not found: ID does not exist" containerID="61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e" Sep 17 18:21:29.968280 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.968236 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e"} err="failed to get container status \"61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e\": rpc error: code = NotFound desc = could not find container \"61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e\": container with ID starting with 61e765ee174af7850a3ac0f7d3643e408767038233ce4bbc66b3a62118d6336e not found: ID does not exist" Sep 17 18:21:29.972471 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.972446 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss"] Sep 17 18:21:29.975723 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:29.975702 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e7d45-predictor-685f9d9bbb-mm5ss"] Sep 17 18:21:30.215478 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:21:30.215398 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" path="/var/lib/kubelet/pods/9f7c7a86-c19a-4817-972e-05b890b9e0ed/volumes" Sep 17 18:29:10.355655 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:10.355572 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg"] Sep 17 18:29:10.358069 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:10.355853 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" containerID="cri-o://44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d" gracePeriod=30 Sep 17 18:29:10.358069 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:10.355899 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kube-rbac-proxy" containerID="cri-o://68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb" gracePeriod=30 Sep 17 18:29:10.866648 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:10.866611 2550 generic.go:356] "Generic (PLEG): container finished" podID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerID="68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb" exitCode=2 Sep 17 18:29:10.866812 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:10.866677 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" event={"ID":"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a","Type":"ContainerDied","Data":"68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb"} Sep 17 18:29:13.391215 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.391187 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:29:13.499256 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.499165 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-rbac-proxy-sar-config\") pod \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " Sep 17 18:29:13.499429 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.499263 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vl5v\" (UniqueName: \"kubernetes.io/projected/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-api-access-4vl5v\") pod \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " Sep 17 18:29:13.499429 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.499298 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-proxy-tls\") pod \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\" (UID: \"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a\") " Sep 17 18:29:13.499615 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.499585 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" (UID: "1f44c675-ab1a-4cc8-80d0-6bbebdcee50a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 18:29:13.501283 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.501255 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-api-access-4vl5v" (OuterVolumeSpecName: "kube-api-access-4vl5v") pod "1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" (UID: "1f44c675-ab1a-4cc8-80d0-6bbebdcee50a"). InnerVolumeSpecName "kube-api-access-4vl5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 18:29:13.501391 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.501328 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" (UID: "1f44c675-ab1a-4cc8-80d0-6bbebdcee50a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 18:29:13.600124 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.600074 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vl5v\" (UniqueName: \"kubernetes.io/projected/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-api-access-4vl5v\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:29:13.600124 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.600121 2550 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-proxy-tls\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:29:13.600124 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.600133 2550 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-40.ec2.internal\" DevicePath \"\"" Sep 17 18:29:13.880805 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.880718 2550 generic.go:356] "Generic (PLEG): container finished" podID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerID="44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d" exitCode=0 Sep 17 18:29:13.880805 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.880773 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" Sep 17 18:29:13.881073 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.880777 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" event={"ID":"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a","Type":"ContainerDied","Data":"44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d"} Sep 17 18:29:13.881073 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.880895 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg" event={"ID":"1f44c675-ab1a-4cc8-80d0-6bbebdcee50a","Type":"ContainerDied","Data":"5d31f2afeb0461ee17c2aa0084e604602089c0aea4adad320cd8d2154255158b"} Sep 17 18:29:13.881073 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.880923 2550 scope.go:117] "RemoveContainer" containerID="68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb" Sep 17 18:29:13.889875 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.889854 2550 scope.go:117] "RemoveContainer" containerID="44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d" Sep 17 18:29:13.897335 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.897316 2550 scope.go:117] "RemoveContainer" containerID="68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb" Sep 17 18:29:13.897671 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:29:13.897653 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb\": container with ID starting with 68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb not found: ID does not exist" containerID="68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb" Sep 17 18:29:13.897721 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.897680 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb"} err="failed to get container status \"68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb\": rpc error: code = NotFound desc = could not find container \"68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb\": container with ID starting with 68622cd5adda45f57914d05db4f953673415a80eefb30d7023765d3cdb4a4dcb not found: ID does not exist" Sep 17 18:29:13.897721 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.897697 2550 scope.go:117] "RemoveContainer" containerID="44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d" Sep 17 18:29:13.897923 ip-10-0-135-40 kubenswrapper[2550]: E0917 18:29:13.897908 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d\": container with ID starting with 44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d not found: ID does not exist" containerID="44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d" Sep 17 18:29:13.897968 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.897937 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d"} err="failed to get container status \"44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d\": rpc error: code = NotFound desc = could not find container \"44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d\": container with ID starting with 44b7b4d5a295a60f7491454575cf2569c4cd6203f6e2e60288be22b9c0639c7d not found: ID does not exist" Sep 17 18:29:13.902179 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.902159 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg"] Sep 17 18:29:13.905708 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:13.905685 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-85bfa-predictor-74c5d488b8-xfmfg"] Sep 17 18:29:14.220404 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:14.220356 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" path="/var/lib/kubelet/pods/1f44c675-ab1a-4cc8-80d0-6bbebdcee50a/volumes" Sep 17 18:29:44.642482 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:44.642446 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g7gkl_1eca5fda-05d1-43bf-8b60-a65f9480bcc3/global-pull-secret-syncer/0.log" Sep 17 18:29:44.804103 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:44.804071 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-p6j48_416dd6dc-3054-408c-bbff-e1bc31acc67d/konnectivity-agent/0.log" Sep 17 18:29:44.872785 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:44.872732 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-40.ec2.internal_323a5990fa2f22ac49cb118e6b347864/haproxy/0.log" Sep 17 18:29:49.024066 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.024029 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8e1c434b-dcac-4386-aaf2-4d8e3273de33/alertmanager/0.log" Sep 17 18:29:49.072558 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.072507 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8e1c434b-dcac-4386-aaf2-4d8e3273de33/config-reloader/0.log" Sep 17 18:29:49.116488 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.116460 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8e1c434b-dcac-4386-aaf2-4d8e3273de33/kube-rbac-proxy-web/0.log" Sep 17 18:29:49.139336 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.139311 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8e1c434b-dcac-4386-aaf2-4d8e3273de33/kube-rbac-proxy/0.log" Sep 17 18:29:49.161378 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.161348 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8e1c434b-dcac-4386-aaf2-4d8e3273de33/kube-rbac-proxy-metric/0.log" Sep 17 18:29:49.183759 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.183731 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8e1c434b-dcac-4386-aaf2-4d8e3273de33/prom-label-proxy/0.log" Sep 17 18:29:49.211685 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.211657 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8e1c434b-dcac-4386-aaf2-4d8e3273de33/init-config-reloader/0.log" Sep 17 18:29:49.368215 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.368131 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d4b6d4d65-62sjd_cde5a7cc-f485-49aa-8fca-06eaf2b5b6f3/metrics-server/0.log" Sep 17 18:29:49.397324 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.397289 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-dwk4g_8bca4d02-f77a-40e6-9f71-7f05742facd6/monitoring-plugin/0.log" Sep 17 18:29:49.436229 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.436200 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ksjc_e4ce6dba-083e-4b97-b623-afe19826b0fe/node-exporter/0.log" Sep 17 18:29:49.458673 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.458642 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ksjc_e4ce6dba-083e-4b97-b623-afe19826b0fe/kube-rbac-proxy/0.log" Sep 17 18:29:49.492952 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.492928 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4ksjc_e4ce6dba-083e-4b97-b623-afe19826b0fe/init-textfile/0.log" Sep 17 18:29:49.694430 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.694407 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ns2kw_2db7991c-a774-40cd-8aa4-57a753d33734/kube-rbac-proxy-main/0.log" Sep 17 18:29:49.716871 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.716848 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ns2kw_2db7991c-a774-40cd-8aa4-57a753d33734/kube-rbac-proxy-self/0.log" Sep 17 18:29:49.736353 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:49.736327 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-ns2kw_2db7991c-a774-40cd-8aa4-57a753d33734/openshift-state-metrics/0.log" Sep 17 18:29:52.554495 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:52.554468 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-xq9j8_fa6ce264-24e7-4a2c-8209-1ccff3030d18/download-server/0.log" Sep 17 18:29:53.209962 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.209927 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9"] Sep 17 18:29:53.210252 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210238 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210254 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210265 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kube-rbac-proxy" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210270 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kube-rbac-proxy" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210278 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210283 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210289 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kube-rbac-proxy" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210294 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kube-rbac-proxy" Sep 17 18:29:53.210303 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210303 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kube-rbac-proxy" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210309 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kube-rbac-proxy" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210316 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210321 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210369 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kserve-container" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210376 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c615fa2-4831-444f-a916-0dfd927210ed" containerName="kube-rbac-proxy" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210382 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kube-rbac-proxy" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210388 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kube-rbac-proxy" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210392 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f44c675-ab1a-4cc8-80d0-6bbebdcee50a" containerName="kserve-container" Sep 17 18:29:53.210607 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.210398 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f7c7a86-c19a-4817-972e-05b890b9e0ed" containerName="kserve-container" Sep 17 18:29:53.213693 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.213675 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.216131 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.216112 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f44dh\"/\"kube-root-ca.crt\"" Sep 17 18:29:53.217083 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.217055 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f44dh\"/\"openshift-service-ca.crt\"" Sep 17 18:29:53.217191 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.217107 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f44dh\"/\"default-dockercfg-mbd8r\"" Sep 17 18:29:53.223986 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.223963 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9"] Sep 17 18:29:53.326868 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.326835 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-proc\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.326868 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.326869 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-podres\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.327087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.326887 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-lib-modules\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.327087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.326976 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-sys\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.327087 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.327003 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjswc\" (UniqueName: \"kubernetes.io/projected/4d59d7bf-89ea-4106-83a2-e67cac94512f-kube-api-access-gjswc\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.427779 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427740 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-proc\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.427779 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427782 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-podres\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.427953 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427804 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-lib-modules\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.427953 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427856 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-sys\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.427953 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427874 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-proc\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.427953 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427884 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjswc\" (UniqueName: \"kubernetes.io/projected/4d59d7bf-89ea-4106-83a2-e67cac94512f-kube-api-access-gjswc\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.427953 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427936 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-podres\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.428113 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427961 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-sys\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.428113 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.427992 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d59d7bf-89ea-4106-83a2-e67cac94512f-lib-modules\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.437452 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.437429 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjswc\" (UniqueName: \"kubernetes.io/projected/4d59d7bf-89ea-4106-83a2-e67cac94512f-kube-api-access-gjswc\") pod \"perf-node-gather-daemonset-ld7h9\" (UID: \"4d59d7bf-89ea-4106-83a2-e67cac94512f\") " pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.523702 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.523609 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:53.792613 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:53.792582 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9"] Sep 17 18:29:53.794972 ip-10-0-135-40 kubenswrapper[2550]: W0917 18:29:53.794948 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4d59d7bf_89ea_4106_83a2_e67cac94512f.slice/crio-5016bade0fdda073232c0e48a15d181c8eda42aa64976ee1b625dbf7178f4068 WatchSource:0}: Error finding container 5016bade0fdda073232c0e48a15d181c8eda42aa64976ee1b625dbf7178f4068: Status 404 returned error can't find the container with id 5016bade0fdda073232c0e48a15d181c8eda42aa64976ee1b625dbf7178f4068 Sep 17 18:29:54.022298 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.022273 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6f52j_fe73488a-e40c-4139-bd07-734d7e7b6cb7/dns/0.log" Sep 17 18:29:54.041589 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.041555 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6f52j_fe73488a-e40c-4139-bd07-734d7e7b6cb7/kube-rbac-proxy/0.log" Sep 17 18:29:54.052001 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.051917 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" event={"ID":"4d59d7bf-89ea-4106-83a2-e67cac94512f","Type":"ContainerStarted","Data":"e419bd4dc9095383cb51afe15ccb2aa903f590ec7956d40ff406fe58e2b983e7"} Sep 17 18:29:54.052001 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.051977 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" event={"ID":"4d59d7bf-89ea-4106-83a2-e67cac94512f","Type":"ContainerStarted","Data":"5016bade0fdda073232c0e48a15d181c8eda42aa64976ee1b625dbf7178f4068"} Sep 17 18:29:54.052236 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.052024 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:29:54.070479 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.070420 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" podStartSLOduration=1.070403484 podStartE2EDuration="1.070403484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:54.06912114 +0000 UTC m=+2940.369688694" watchObservedRunningTime="2026-09-17 18:29:54.070403484 +0000 UTC m=+2940.370971027" Sep 17 18:29:54.208357 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.208321 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd6zm_573ca411-6849-4ca8-885f-d3985df41a88/dns-node-resolver/0.log" Sep 17 18:29:54.689464 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.689428 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bc47694cb-555dv_54af9832-412c-4a7d-9503-a7f84b6aafad/registry/0.log" Sep 17 18:29:54.737320 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:54.737291 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dc42q_173230ea-68b4-4625-9d03-bc0eaff8a310/node-ca/0.log" Sep 17 18:29:55.822614 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:55.822584 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nb9lq_cb976eb3-1765-41b5-a659-2e3b4c46c7c1/serve-healthcheck-canary/0.log" Sep 17 18:29:56.387235 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:56.387210 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrbjn_64162815-9899-48b9-b6e2-550fd15cff5e/kube-rbac-proxy/0.log" Sep 17 18:29:56.406206 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:56.406176 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrbjn_64162815-9899-48b9-b6e2-550fd15cff5e/exporter/0.log" Sep 17 18:29:56.425926 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:56.425894 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zrbjn_64162815-9899-48b9-b6e2-550fd15cff5e/extractor/0.log" Sep 17 18:29:58.657446 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:58.657411 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5c5cdf4877-mmpxz_4d8d9d7e-e4c5-4310-abc8-80c6dd6680d4/manager/0.log" Sep 17 18:29:58.695589 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:58.695555 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-sx7zn_26d2722e-ee53-4ea5-9f98-dbd9772897d8/manager/0.log" Sep 17 18:29:58.861971 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:29:58.861916 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-4bjvs_8b079cd9-39b5-4cb2-adea-86276d5ab0b8/manager/0.log" Sep 17 18:30:00.066741 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:00.066716 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f44dh/perf-node-gather-daemonset-ld7h9" Sep 17 18:30:04.688885 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.688849 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zxrbf_c4b5481b-65d4-49a4-827f-73254a600f11/kube-multus-additional-cni-plugins/0.log" Sep 17 18:30:04.708037 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.708007 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zxrbf_c4b5481b-65d4-49a4-827f-73254a600f11/egress-router-binary-copy/0.log" Sep 17 18:30:04.726626 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.726603 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zxrbf_c4b5481b-65d4-49a4-827f-73254a600f11/cni-plugins/0.log" Sep 17 18:30:04.745169 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.745086 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zxrbf_c4b5481b-65d4-49a4-827f-73254a600f11/bond-cni-plugin/0.log" Sep 17 18:30:04.763501 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.763468 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zxrbf_c4b5481b-65d4-49a4-827f-73254a600f11/routeoverride-cni/0.log" Sep 17 18:30:04.784762 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.784737 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zxrbf_c4b5481b-65d4-49a4-827f-73254a600f11/whereabouts-cni-bincopy/0.log" Sep 17 18:30:04.806663 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.806637 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zxrbf_c4b5481b-65d4-49a4-827f-73254a600f11/whereabouts-cni/0.log" Sep 17 18:30:04.844442 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.844418 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cmxnt_6847f4d4-2b04-47ca-82ee-399ef7517d45/kube-multus/0.log" Sep 17 18:30:04.959846 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.959816 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gg8ft_a7996a81-8ee5-41db-a2a0-e3a1f6b13367/network-metrics-daemon/0.log" Sep 17 18:30:04.976441 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:04.976415 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gg8ft_a7996a81-8ee5-41db-a2a0-e3a1f6b13367/kube-rbac-proxy/0.log" Sep 17 18:30:05.916481 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:05.916453 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/ovn-controller/0.log" Sep 17 18:30:05.932200 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:05.932177 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/ovn-acl-logging/0.log" Sep 17 18:30:05.956183 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:05.956154 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/kube-rbac-proxy-node/0.log" Sep 17 18:30:05.978820 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:05.978795 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/kube-rbac-proxy-ovn-metrics/0.log" Sep 17 18:30:05.997075 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:05.997044 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/northd/0.log" Sep 17 18:30:06.016477 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:06.016430 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/nbdb/0.log" Sep 17 18:30:06.035229 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:06.035205 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/sbdb/0.log" Sep 17 18:30:06.090724 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:06.090695 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fzxzd_ddaaca48-ac39-4e85-8d85-f78b1467d084/ovnkube-controller/0.log" Sep 17 18:30:07.357571 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:07.357541 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dm5ls_485f436b-3de3-4aeb-aec2-702f81e96c0a/network-check-target-container/0.log" Sep 17 18:30:08.309353 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:08.309321 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-w2jj8_3bcb8968-7f79-4c45-aab8-bc441842d62d/iptables-alerter/0.log" Sep 17 18:30:09.025024 ip-10-0-135-40 kubenswrapper[2550]: I0917 18:30:09.024991 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-rwdwk_970e8def-d389-4168-a5d2-17cb0cc42962/tuned/0.log"